Distributed Control of a Heterogeneous Video Surveillance Network

ABSTRACT

A surveillance video broker arbitrates access by multiple clients to multiple surveillance video sources. Both clients and sources register with the broker. Each source independently specifies respective clients permitted real-time access to its video and conditions of access, if any. Preferably, the video source is a local surveillance domain having one or more cameras, one or more sensors, and a local controller, the source specifying clients or client groups permitted access, and independently specifying conditions of access for each client or client group, where conditions may include scheduled events, non-scheduled events, such as alarms or emergencies, and/or physical proximity. The broker automatically authorizes real-time access according to pre-specified conditions. Preferably, the broker can also arbitrate alert notifications to the clients based on pre-specified notification criteria.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation of pending U.S. patent application Ser. No.13/751,013, filed Jan. 25, 2013, entitled “Distributed Control of aHeterogeneous Video Surveillance Network”, which is a continuation ofpending U.S. patent application Ser. No. 13/675,349, filed Nov. 13,2012, entitled “Distributed Control of a Heterogeneous VideoSurveillance Network”, both of which are herein incorporated byreference. This application claims priority under 35 U.S.C. §120 of U.S.patent application Ser. No. 13/751,013, filed Jan. 25, 2013, and of U.S.patent application Ser. No. 13/675,349, filed Nov. 13, 2012.

The present application is also related to the following commonlyassigned U.S. patent applications, each of which are herein incorporatedby reference:

Ser. No. 13/675,391, filed Nov. 13, 2012, which issued Jan. 13, 2015 asU.S. Pat. No. 8,934,754, entitled “Providing Emergency Access toSurveillance Video”;

Ser. No. 13/675,429, filed Nov. 13, 2012, which issued May 26, 2015, asU.S. Pat. No. 9,041,812, entitled “Automated Authorization to AccessSurveillance Video Based on Pre-Specified Events”;

Ser. No. 13/751,023, filed Jan. 25, 2013, which issued Jun. 30, 2015, asU.S. Pat. No. 9,071,807, entitled “Providing Emergency Access toSurveillance Video”; and

Ser. No. 13/751,029, filed Jan. 25, 2013, which issued Nov. 17, 2015, asU.S. Pat. No. 9,191,632, entitled “Automated Authorization to AccessSurveillance Video Based on Pre-Specified Events”.

FIELD OF THE INVENTION

The present invention relates to digital data processing, and inparticular to digital data systems for monitoring and control of a setof video surveillance devices including multiple video cameras atdifferent locations.

BACKGROUND

In the latter half of the twentieth century, there began a phenomenonknown as the information revolution. While the information revolution isa historical development broader in scope than any one event or machine,no single device has come to represent the information revolution morethan the digital electronic computer. The development of computersystems has surely been a revolution. Each year, computer systems growfaster, store more data, and provide more applications to their users.At the same time, the cost of computing resources has consistentlydeclined, so that information which was too expensive to gather, storeand process a few years ago, is no economically feasible to manipulatevia computer. The reduced cost of information processing drivesincreasing productivity in a snowballing effect, because productdesigns, manufacturing processes, resource scheduling, administrativechores, and many other tasks, are made more efficient.

Among the applications of digital data processing technology is thecapture, storage, transmission and other processing of images, both instill and motion video forms. A large variety of applications nowprocess images is some form or another. So large is the demand for imageprocessing that standard computer systems are now equipped with specialprocessors and other hardware for that purpose, and a substantial amountof digital data storage is devoted to the storage of still and motionvideo images.

One particular application for the processing of images is videosurveillance. Early video surveillance systems were essentially analogin design, typically capturing video on one or more analog video camerasand conveying a video feed by wired connection (e.g. coaxial cable) to acentral monitoring station, at which security personnel could monitorand/or record the multiple feeds, and switch among the various feeds.These systems were typically relatively expensive, were limited inrange, and required trained personnel for monitoring and maintenance. Asa result, they were typically used only by sophisticated enterprises,and then only where a strong need was established which would justifythe expense.

Digital imaging and digital networking have generally supplanted theseanalog designs and reduced the cost of processing video information. Forexample, it is now possible to capture video images using inexpensivedigital video cameras, to transmit video via low-cost limited rangewireless networks, wired LANs, or at greater ranges via the Internet orother networks, to display video on mass-produced consumer devices suchas desktop personal computers, laptops, notebook computers, smartphones, and other portable digital devices, and to store large amountsof video on relatively inexpensive mass storage devices.

The capabilities of modern digital data processing and imaging deviceshave placed sophisticated video surveillance technology within practicalreach of the homeowner or small business. For example, off-the-shelfsystems may now be purchased by the homeowner or small business ownerfor monitoring an area with one or several cameras, transmitting videofrom the camera via a local wireless connection such as an IEEE 802.11“WiFi” connection, collecting the video in a local computer system, andaccessing the video, either in real time or after the fact, from aremote digital data device via the Internet.

The relatively low cost of modern digital surveillance technology,particularly small, off-the-shelf surveillance systems intended forhomeowners and small businesses, has greatly increased the number ofsuch surveillance systems in use. Furthermore, it can be expected thatas public awareness of such systems increases, improvements are made totheir capabilities and ease of use, and additional cost savings arerealized, many more such systems will be installed.

The reduced cost of individual digital video components, such as digitalvideo cameras and transmission hardware, has led to some far reachingproposals for digital surveillance. In some cases, it has been proposedthat entire cities or neighborhoods within cities be placed undersurveillance by some central authority, for use by police, firefightersand/or other emergency response personnel. However, although individualcomponents of a video surveillance system are relatively inexpensive,when these components are multiplied by the number of componentsnecessary to place large regions of a city under surveillance, the costof such a system is very substantial. This cost is a major factorconstraining the deployment of widescale surveillance systems. The factthat such systems are under serious consideration, and in some caseshave even been deployed (although not necessarily on a comprehensivebasis), is evidence of the perceived need for such systems.

In addition to the cost of the hardware, its installation andmaintenance, there are other barriers to the deployment of comprehensivesurveillance systems. A municipality or similar government agency hasunrestricted access to only a portion of the area which it may bedesirable to place under surveillance. Much of the subject area will beunder the control of various private entities, any of which may balk atgiving a government agency unrestricted surveillance rights over itsproperty. It is taken for granted that a homeowner will be reluctant togrant surveillance rights to areas within a home, but even a privatebusiness will often be reluctant to allow unrestricted governmentalsurveillance over areas under its control, whether for use by itsemployees, customers and/or the general public.

While private businesses and individuals are understandably reluctant toallow unrestricted surveillance of their property by others, there arecircumstances in which they may not only be willing, but desirous, ofsuch surveillance. A prime example of such a circumstance is theoccurrence of an emergency requiring the intervention of emergencyresponse personnel, such as a fire, a criminal act, or a medicalemergency. In such an event, access to surveillance video in real timeby dispatchers, emergency responders, and the like, may improve thespeed and efficacy of the response, outweighing any concerns aboutprivacy. There are other circumstances, although less compelling, inwhich surveillance by others may be deemed desirable or acceptablesimply because privacy is of minimal concern. For example, a privatebusiness may be willing to grant surveillance rights over an area at atime of day during which the business is closed and nobody is authorizedto be on the premises.

A need exists, not necessarily recognized, for improved methods andsystems for managing and accessing surveillance video, and particularlyfor conditionally accessing, by others, surveillance video under thecontrol of a property owner, tenant, or the like.

SUMMARY

A video broker arbitrates access by multiple parties to multiplesurveillance video sources. Each of multiple parties (acting as clients)registers with the video broker to obtain access to multiple videosources. Each of multiple surveillance video source controlling entitiesregisters with the video broker and independently specifies one or morerespective clients permitted real-time access to its video andconditions of access, if any, each client having independentlyspecifiable access rights.

In one inventive aspect, one or more clients are granted conditionalaccess to video by one or more surveillance video source controllingentities by defining respective one or more pre-specified conditionsunder which access is to be granted, the access being automaticallygranted if the corresponding one or more pre-specified conditions aremet.

In another inventive aspect, the independent grant of video accessrights by multiple video source controlling entities to multiple clientsrequires no one-to-one correspondence between source and client or setsof clients, and a first client is granted access to multiple sources butfewer than all of the sources, while a second client is granted accessto some but not all of the sources to which the first client has access.

In another inventive aspect, each of multiple video source controllingentities further independently specifies rights by clients to receivealert notifications generated by systems under its control, and clientsindependently specify which alerts they will receive. Alertnotifications are sent to the video broker, which determines whichclients will receive notifications based on pre-existing specifications,and forwards notifications accordingly.

In a preferred embodiment incorporating these various aspects, asurveillance video source controlling entity (such as a property owneror tenant) collects surveillance video from one or more cameras in alocal surveillance domain control system under its control, which may bea dedicated security system or a general purpose computer system whichis used for various tasks, video surveillance being only one of thetasks. Preferably, surveillance video data is stored in the surveillancedomain control system. Video may be transmitted in real time from thecameras to the surveillance domain control system via any of variousknown protocols using local wired or wireless connections. Thesurveillance domain control system has the capability to detect one ormore alert events, either through input received from separate sensors(such as motion detectors, perimeter sensors, smoke and heat detectors,auditory sensors, etc.), or by analysis of the video itself. Thesurveillance domain control system communicates with a remotely locatedsurveillance video broker over a network, which is preferably theInternet. The surveillance domain control system registers itself withthe surveillance video broker, providing such information as may beuseful to characterize the domain, such as the number and location ofcameras under its control, data protocol used for video transmission,and so forth. In one or more embodiments, the surveillance domain maypre-specify to the broker the clients to receive notification ofdetected events, and the pre-defined conditions under which video fromthese cameras can be accessed by clients. In one or more alternativeembodiments, the surveillance domain may specify clients to receivenotification of detected events at the time it notifies the broker ofthe detected event, and/or specify pre-defined conditions to the brokerwhen access is requested, or itself determine whether any pre-definedconditions are met. These pre-defined conditions could include any oneor more of: (a) the identity or nature of the client desiring access;(b) an event state relating to recently occurring non-scheduled events,such as alert events; (c) a time of day, day of week, or other scheduledevent; and (d) a physical proximity of a requesting client or agent ofthe client to the area under surveillance. Clients also register withthe surveillance video broker independently of the surveillance domaincontrol systems. Each client may independently provide in advancecriteria specifying the alert events for which the respective clientwishes to receive notice, or alternatively, filter any notificationswithin the client itself. Upon the detection of an alert event in thesurveillance domain control system, an alert message is sent to thesurveillance video broker. The broker compares the alert eventparameters with any conditions placed on alert notification by thesurveillance domain control system, as well as criteria specified byclients for receipt of alert notification (if any), to determine whichclient(s) should receive notification of the alert, and forwards analert message to each client entitled to notification. A client canrequest real-time access to a camera under control of the localsurveillance domain control system by communicating with thesurveillance video broker (preferably over the Internet) and providingsuch identifying information as may be required. If the pre-definedconditions for access the video are met, access is automatically grantedto the requesting client, which then receives transmission of therequested video in real time. For reasons of bandwidth, in oneembodiment video is transmitted from the surveillance domain controlsystem to a separate video distribution service having a high bandwidthnetwork connection, and from there to as many clients as have requestedand been granted access to it. Video could alternatively be transmitteddirectly from the surveillance domain control system to each requestingclient. It will be understood that the preferred embodiment and certainalternatives described above are only some of the possible embodiments,and many additional variations are possible.

In one embodiment, a public authority may declare a state of emergency.If the surveillance video source controlling entity has previouslyauthorized access to its video by selective persons, such as publicauthority emergency responders, under a “declared emergency” condition,access is automatically granted to such persons, without requiringadditional authorization from the source controlling entity. Allaccesses are logged to provide an audit trail and inhibit excessive useof the “declared emergency”.

By providing a flexible distributed mechanism for controlling aheterogenous set of surveillance video sources and providing alerts andvideo to appropriate clients as disclosed herein, surveillance video iseasier to share, greater access to surveillance video can be obtainedfor use by emergency response personnel or other parties having alegitimate interest in it, and private parties do not have to undulyrelinquish control over their own video sources. Furthermore, aflexible, distributed mechanism for controlling video surveillance asdisclosed herein is readily adaptable to integration of pre-existingvideo surveillance systems into a larger surveillance network, thusreducing need for new installations and associated costs.

The details of the present invention, both as to its structure andoperation, can best be understood in reference to the accompanyingdrawings, in which like reference numerals refer to like parts, and inwhich:

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 is a high-level conceptual illustration of a networkedsurveillance system environment supporting distributed control ofsurveillance video, according to the preferred embodiment of the presentinvention.

FIGS. 2A-2C illustrate several different variations of the majorhardware components of exemplary surveillance domains, FIG. 2Arepresenting a first exemplary surveillance domain, in the form of alegacy analog video surveillance domain, FIG. 2B representing a secondexemplary surveillance domain, in the form of a surveillance domainusing digital wireless technology, and FIG. 2C representing a thirdexemplary surveillance domain, in the form of a mobile surveillancedomain, in accordance with one or more preferred embodiments.

FIG. 3 is a high-level block diagram of the major hardware components ofa representative general purpose computer system which could be used toperform the role of any of several functional elements, according to thepreferred embodiment.

FIG. 4 is a conceptual illustration of the major software components ofa computer system of FIG. 3 configured as a surveillance video brokercomputer system, according to the preferred embodiment.

FIG. 5 is a conceptual illustration of a clients table in a video brokerdatabase, according to the preferred embodiment.

FIG. 6 is a conceptual illustration of a client groups table in a videobroker database, according to the preferred embodiment.

FIG. 7 is a conceptual illustration of a group membership table in avideo broker database, according to the preferred embodiment.

FIG. 8 is a conceptual illustration of a surveillance domains table in avideo broker database, according to the preferred embodiment.

FIG. 9 is a conceptual illustration of a cameras table in a video brokerdatabase, according to the preferred embodiment.

FIG. 10 is a conceptual illustration of an alert notifications table ina video broker database, according to the preferred embodiment.

FIG. 11 is a conceptual illustration of an alert filters table in avideo broker database, according to the preferred embodiment.

FIG. 12 is a conceptual illustration of a video authorizations table ina video broker database, according to the preferred embodiment.

FIG. 13 is a conceptual illustration of an events table in a videobroker database, according to the preferred embodiment.

FIG. 14 is a conceptual illustration of a video accesses table in avideo broker database, according to the preferred embodiment.

FIG. 15 is a flow diagram illustrating at a high level the operation ofa surveillance domain service function executing in a surveillance videobroker, according to the preferred embodiment.

FIG. 16 is a flow diagram illustrating in greater detail a process forhandling an alert notification request, according to the preferredembodiment.

FIGS. 17A and 17B (herein collectively referred to as FIG. 17) are aflow diagram illustrating at a high level the operation of a clientservice function executing in surveillance video broker, according tothe preferred embodiment.

FIGS. 18A and 18B (herein collectively referred to as FIG. 18) are aflow diagram illustrating in greater detail a process for handling arequest by a client to access surveillance video, according to thepreferred embodiment.

FIG. 19 is a flow diagram illustrating in greater detail a process forhandling a request by a client to declare a state of emergency,according to the preferred embodiment.

DETAILED DESCRIPTION

Referring to the Drawing, wherein like numbers denote like partsthroughout the several views, FIG. 1 is a high-level conceptualrepresentation of a networked surveillance system environment 100supporting distributed control of surveillance video, according to oneor more preferred embodiments of the present invention. The environmentincludes a network (or collection of networks) 101. In the preferredembodiment network 101 includes the Internet, and may optionally beaugmented by one or more cell phone networks for transmitting data toportable devices, it being understood that other networks or collectionsof networks could alternatively be used. Multiple surveillance domains102A-D (herein generically referred to as feature 102) are each coupledto the network, each surveillance domain including one or morerespective video cameras and serving as a source of surveillance video.Surveillance domains 102 may be of varying size, age and configuration,and are not owned or controlled by any common entity. Multiple clientdevices 104A-E (herein generically referred to as feature 104) are alsocoupled to the network. A client device may be any digital data devicecapable of communicating with network 101 and receiving video datatherefrom originating in one of surveillance domains 102 for display toa user of the client device. For example, a client device could be anyof: a general purpose digital computer of any type, from a largemulti-user computer serving multiple attached terminals or workstations,to a single-user desktop or laptop personal computer; a personal digitalassistant or similar portable device; a smart phone; an on-board vehiclecomputer and display; a special purpose security monitoring device; andso forth.

Surveillance system environment 100 further includes a surveillancevideo broker 103 coupled to network 101. Surveillance video broker 103serves as a proxy for the passing of messages between surveillancedomains 102 and clients 104, and manages access by clients 104 to videoproduced by surveillance domains 102 in accordance with accessdirectives provided by the surveillance domains, as explained in greaterdetail herein. In a preferred embodiment, surveillance systemenvironment further includes optional video distribution service 105coupled to network 101. Video distribution service 105 receives videodata produced by multiple surveillance domains 102, optionally storesthe data, and re-transmits it to clients 104 as authorized bysurveillance video broker 103. Surveillance video broker 103 and videodistribution service 105 are shown in FIG. 1 as separate entities forpurposes of representing the division of function performed by thesurveillance video broker and the video distribution service. Broker 103and distribution service 105 could in fact be separate physical systemsat remote locations which communicate with each other over the Internetor other network, or could be separate physical systems at a commonlocation which communicate with each other over a local area network orother local communications channel, or could be a single physical systemwhich performs the functions of broker 103 as well as distributionservice 105. Furthermore, broker 103 and distribution service 105 couldbe under the control of different entities or a common entity.

Although FIG. 1 represents a particular number of surveillance domains102, client devices 104, and a single broker 103 and video distributionservice 105, for illustrative purposes, it will be appreciated that thenumber of such device could vary, that the number of surveillancedomains and client devices would typically be much larger, and thatthere may be multiple brokers 103 and/or distribution services 105 toprovide redundancy and/or additional capacity

FIGS. 2A-2C illustrate, in greater detail than shown in FIG. 1, severaldifferent variations of the major hardware components of exemplarysurveillance domains 102, in accordance with one or more preferredembodiments. Common to each of these exemplary surveillance domains isat least one video camera 201A-I (herein referred to generically asfeature 201) and a local surveillance domain control system 202A-C(herein generically referred to as feature 202) in communication withnetwork 101 and capable of receiving video output from the at least onecamera 201.

FIG. 2A represents a first exemplary surveillance domain, in the form ofa legacy surveillance domain, in which each camera 201A-D produces ananalog video signal and transmits it via a wired connection (e.g.,coaxial cable) to a splitter/switch hub 205. Hub 205 has the capabilityto output a signal (via wired connection) to any of one or more analogvideo tape recorders 206A-D, and/or local surveillance domain controlsystem 202A, and can output multiple copies of the same signal. Hub 205may optionally output signals to local viewing monitors or other devices(not shown). Control system 202A is represented as a multi-user computersystem, having multiple attached workstation terminals 203A, 203B.Control system 202A receives one or more analog video signals from hub205, and converts the analog signal to an appropriate digital videoformat using embedded analog-to-digital video conversion hardware.Multiple sensors 204A-C, represented as wired perimeter sensors 204A,204B which sense a door being opened and closed, and wired smoke/heatsensor 204C for sensing a fire, provide alarm signals to alarmcontroller 207. Alarm controller communicates an alarm signal to controlsystem 202A over any appropriate communication link (e.g., a serialcommunication line), enabling control system 202A to detect alarmconditions. System 202A is coupled to network 101 via a wiredconnection. In such a legacy system, the cameras 201, hub 205, recorders206, sensors 204 and/or alarm 207 could have been installed earlier thancontrol system 202.

FIG. 2B represents a second exemplary surveillance domain, in the formof a surveillance domain using more modern digital wireless technology,in which each camera 201E-H produces a digital video signal andtransmits it via a local wireless transmission (e.g., an IEEE 802.11“WiFi” transmission) from the camera to wireless router 208. Wirelessrouter is coupled to local surveillance domain control system 202B,which is represented in this case as a stand-alone single user desktopcomputer system, via a wired link, although the link could alternativelybe wireless. Sensors 204D-E, represented as motion detector 204D andsmoke/heat sensor 204E, are also in communication with router 208 vialocal wireless connection, allowing them to communicate alarm conditionsto control system 202B. Router 208 is also connected to network 101,providing a means for control system 202B to communicated with thenetwork.

FIG. 2C represents a third exemplary surveillance domain, in the formmobile surveillance domain in which a camera 2011 as well as a localsurveillance domain control system 202C is mounted in a moving object,such as a motor vehicle. Camera 2011 is preferably a digital videocamera, which may be directly coupled to control system 202C via a wiredconnection as shown, or may communicate by wireless connection. Themotor vehicle preferably includes a global positioning system (GPS) 209coupled to (or integrated with) control system 202C, allowing thelocation of the vehicle to be tracked. The vehicle may optionallyinclude other sensors or devices (not shown) to provide input to controlsystem 202C useful for operating camera 201 or generating alarms, suchas directional sensors, speed sensors, collision detectors, and soforth. Control system 202C communicates via wireless connection withnetwork 101, and may either contain an integrated wireless transceiveror be coupled to a separate wireless transceiver (not shown) mountedwithin the motor vehicle. Although the moving object is represented inFIG. 2C as a motor vehicle, a moving object need not be a motor vehicleor an object approximating its size, and could alternatively be a smallhandheld device, such as a smart phone containing integrated camera,control system, and/or optional devices.

In will be understood that FIGS. 2A-C are intended only to representthree different exemplary surveillance domains 102, and that many othervariations or permutations of surveillance domains and surveillancedomain components are possible in addition to those explicitly disclosedherein. In particular, without limiting the generality of the foregoing,the number and type of video cameras may vary; a single surveillancedomain may contain a mixture of analog and digital video cameras; thenumber and type of sensors may vary; sensors of a type not shown may bepresent, such as glass breakage sensors, temperature sensors, gas leaksensors, water leak sensors, audio sensors, chemical spill sensors,radiation sensors, and so forth; any of various manually-tripped alarmscould be coupled to system 202 to provide an alarm signal, such aswell-known manually operated fire alarms, bank teller or store clerkalarms, and the like; components can communicate via wired or wirelesslinks, and some components may be wired while others are wireless. Whilecontrol system 202 is preferably a general purpose computer systemsuitably programmed to provide the functions described herein, it couldbe any of various digital devices, including portable andspecial-purpose devices, capable of communicating with network 101 andproviding video and other data. Furthermore, control system 202 need notbe a single computer system or device, but may be a collection ofdevices which provide the necessary function and/or provide redundancy.Furthermore, although a single communications link between network 101and control system 202 is shown, there could be multiple link whichcould be redundant or which could be used to provide different types ofdata, e.g. one link used for video data while another is used for alarmsignals and control data. Furthermore, although FIGS. 2A-C represent acommon local control system 202 for controlling video surveillance dataand for receiving sensor data upon which alert conditions are detected,a surveillance domain could alternatively include separate systems forcontrolling the video surveillance data and for detecting andcommunicating alert conditions, which may use a common or separatecommunication links for communicating with network 101.

It will be understood that a networked surveillance system environment100 may contain many surveillance domains each having a differentinternal configuration, and the networked surveillance systemenvironment is therefore not limited to surveillance cameras orsurveillance domains of a particular type.

For clarity of illustration and understanding, the figures herein andaccompanying description represent surveillance domains and clients asseparate entities. However, a client device can be any digital devicewhich performs the function corresponding to a client device asdescribed herein. It would therefore be possible, for example, for localsurveillance domain control system 202, which is preferably a suitableprogrammed general-purpose computer system, to also function as a clientdevice. For example, such a local surveillance domain control systemmight be used as a client device to access surveillance video producedby nearby surveillance domains, or to receive alerts from nearbysurveillance domains.

Each surveillance domain 102 is owned or controlled by a respectivesurveillance video source controlling entity, such as an owner or tenantof property under surveillance. Although one or more of the domains maybe owned or controlled by a respective government agency, there is nocommon ownership or control of the surveillance domains. Specifically,in accordance with the preferred embodiment, at least some of thesurveillance domains are under ownership or control of a private party,such as a business or homeowner (herein “privately-controlledsurveillance domain”), and furthermore, at least some of the sourcecontrolling entities are different. Each video source controllingentity, whether a private party or public or quasi-public agency,independently defines the clients entitled to access its video andconditions for access. The authorized clients may be different fordifferent domains, and the conditions for access need not be the samefor all clients authorized to a single source domain, as explained ingreater detail herein. Furthermore, preferably at least some of theauthorized clients are not the respective video source controllingentity or persons affiliated therewith. Typically, a video sourcecontrolling entity will have unconditional access to the correspondingvideo it controls. Other clients may have unconditional access, but willoften some form of conditional access. For example, access under one ormore pre-specified conditions may be given to public authority emergencyresponders, neighbors, customers, passers-by on the street, and soforth. The surveillance video broker may be owned or controlled by oneof the source controlling entities which controls a surveillance domain,or be a completely separate entity.

FIG. 3 is a high-level block diagram of the major hardware components ofa representative general purpose computer system 300. In the preferredembodiment, surveillance video broker 103, video distribution service105, one or more clients 104, and one or more surveillance domaincontrol systems 202 are physically embodied as respective one or moregeneral purpose computer systems, system 300 being a representation ofany such general purpose computer system.

Computer system 300 includes at least one general-purpose programmableprocessor (CPU) 301 which executes instructions and processes data frommain memory 302. Main memory 302 is preferably a random access memoryusing any of various memory technologies, in which data is loaded fromstorage or otherwise for processing by CPU 301.

One or more communications buses 305 provide a data communication pathfor transferring data among CPU 301, main memory 302 and various I/Ointerface units 311, 312, 313, 314A, 314B, which may also be known asI/O processors (IOPs) or I/O adapters (IOAs). The I/O interface unitssupport communication with a variety of storage and I/O devices. Forexample, terminal interface unit 311 supports the attachment of one ormore user terminals 321-324. Storage interface unit 312 supports theattachment of one or more direct access storage devices (DASD) 325-327(which are typically rotating magnetic disk drive storage devices,although they could alternatively be other devices, including arrays ofdisk drives configured to appear as a single large storage device to ahost). I/O device interface unit 313 supports the attachment of any ofvarious other types of I/O devices, such as printer 328 and fax machine329, it being understood that other or additional types of I/O devicescould be used. Network interface adapters 314A, 314B (herein genericallyreferred to as feature 314) support connections to one or more externalnetworks for communication with one or more other digital devices, andspecifically to network 101 for communication with devices representedin FIG. 1. While two network adapters 314 and network connections areshown, there may be only a single adapter and connection, or there couldbe more than two. Such external networks preferably include theInternet, and may include one or more intermediate networks, such aslocal area networks (not shown), through which communication with theInternet is effected.

It should be understood that FIG. 3 is intended to depict therepresentative major components of general purpose computer system 300at a high level, that individual components may have greater complexitythan represented in FIG. 3, that components other than or in addition tothose shown in FIG. 3 may be present, and that the number, type andconfiguration of such components may vary, and that a complex computersystem will typically have more components than represented in FIG. 3.Several particular examples of such additional complexity or additionalvariations are disclosed herein, it being understood that these are byway of example only and are not necessarily the only such variations.

Although only a single CPU 301 is shown for illustrative purposes inFIG. 3, computer system 300 may contain multiple CPUs, as is known inthe art. Although main memory 302 is shown in FIG. 3 as a singlemonolithic entity, memory 302 may in fact be distributed and/orhierarchical, as is known in the art. E.g., memory may exist in multiplelevels of caches, and these caches may be further divided by function,so that one cache holds instructions while another holds non-instructiondata which is used by the processor or processors. Memory may further bedistributed and associated with different CPUs or sets of CPUs, as isknown in any of various so-called non-uniform memory access (NUMA)computer architectures. Although communications buses 305 are shown inFIG. 3 as a single entity, in fact communications among various systemcomponents is typically accomplished through a complex hierarchy ofbuses, interfaces, and so forth, in which higher-speed paths are usedfor communications between CPU 301 and memory 302, and lower speed pathsare used for communications with I/O interface units 311-314. Buses 305may be arranged in any of various forms, such as point-to-point links inhierarchical, star or web configurations, multiple hierarchical buses,parallel and redundant paths, etc. For example, as is known in a NUMAarchitecture, communications paths are arranged on a nodal basis. Busesmay use, e.g., an industry standard PCI bus, or any other appropriatebus technology. While multiple I/O interface units are shown whichseparate buses 305 from various communications paths running to thevarious I/O devices, it would alternatively be possible to connect someor all of the I/O devices directly to one or more system buses.

Computer system 300 depicted in FIG. 3 has multiple attached terminals321-324, such as might be typical of a multi-user “mainframe” computersystem. Typically, in such a case the actual number of attached devicesis greater than those shown in FIG. 3, although the present invention isnot limited to systems of any particular size. Computer system 300 mayalternatively be a single-user system, typically containing only asingle user display and keyboard input. Furthermore, while the inventionherein is described for illustrative purposes as embodied in a singlecomputer system, the present invention could alternatively beimplemented using a distributed network of computer systems incommunication with one another, in which different functions or stepsdescribed herein are performed on different computer systems.

In the preferred embodiment, broker 103, video distribution service 105,local surveillance domain controllers 202, and at least some clients104, are general purpose computer systems capable of being programmed toexecute a variety of different functions by loading and executingappropriate software. However, any of these devices could alternativelybe special-purpose digital data devices for accomplishing thecorresponding functions. For example, one or more of client devices maybe any of various smart phones or portable devices capable of receivingand displaying video to a user. As a further example, functions of alocal surveillance domain controller 202 may be performed by one or morespecial purpose devices, e.g. alarm signals transmitted by a dedicatedspecial purpose alarm controller while video is transmitted by aseparate device, as might be used in legacy surveillance systems.

While various system components have been described and shown at a highlevel, it should be understood that a typical computer system containsmany other components not shown, which are not essential to anunderstanding of the present invention.

FIG. 4 is a conceptual illustration of the major software components ofcomputer system 300 in memory 302, configured to provide the function ofa surveillance video broker 103, according to the preferred embodiment.Operating system kernel 401 is executable code and state data providingvarious low-level software functions, such as device interfaces,management of memory pages, management and dispatching of multipletasks, etc. as is well-known in the art. In particular, OS kernel 401includes one or more network adapter drivers 402 for handlingcommunications with one or more external networks via network adapters314. A video broker database 403 contains data structures and databasemanagement software for supporting the functions of surveillance videobroker 103. Additionally, one or more software application programscollectively designated a video broker control program 430 executesvarious functions and accesses data in database 403 to provide thesurveillance video broker services described herein.

Video broker database 403 is preferably a structured relational databasecontaining various video broker data structures 404 and a video brokerdatabase manager 420 for managing and accessing the data structures 404.In the preferred embodiment, data structures 404 include an events table405, video authorizations table 406, accesses table 407, notificationstable 408, filters table 409 clients table 410, cameras table 411,surveillance domains table 412, group membership table 413, and groupstable 414, and may contain additional tables (not shown). As is known inthe database art, a database table is a data structure logically in theform of a table having multiple records (also called entries or tuples),each record having at least one, and usually multiple, fields (alsocalled attributes). Although tables 404-414 are data structures whichare logically equivalent to tables, they may be arranged in any suitablestructure known in the database art. Associated with the database tablesare one or more auxiliary data structures 415-417, also sometimesreferred to as metadata (of which three are represented in FIG. 4, itbeing understood that the number of such structures may vary). Auxiliarydata structures characterize the structure of the database and datatherein, and are useful in various tasks involved in databasemanagement, particularly in executing queries against the database.Examples of auxiliary data structures include database indexes,materialized query tables, histograms, index advice record, and soforth, it being understood that other types of metadata may exist.

Clients table 410 contains multiple records, each record correspondingto a respective client 104 in networked surveillance system environment100, which is registered with broker 103 (acting as a server) to accessvideo produced by one or more of the surveillance domains 102 or performadministrative functions. FIG. 5 is a conceptual illustration showing ingreater detail clients table 410 in video broker data 404, according tothe preferred embodiment. Each record 501 in clients table 410 containsa client identifier (ID) 502, a password 503 for accessing the broker,and one or more fields specifying a preferred contact method for alertnotifications. As represented in FIG. 5, these include e-mail flag 504,voice message flag 505, text message flag 506, and instant message flag507, e-mail address 508, and telephone number 509. Flags 504-507indicate, with respect to each method of contact, whether an alertnotification is to be sent using the corresponding method. Multiplemethods of contact might be specified. Mobile flag 510 indicates whetherthe client device is a mobile device In one variation of a preferredembodiment, one or more client records contain location information 511specifying a current location of a client. Such information may be usedto authorized access to video by the client or to filter alertnotifications received by the client. If the client is a mobile client,this location information may be updated periodically by the client. Therecord may contain any of various additional fields useful to identify aclient. For example, additional fields may include a name of a person512, a street address 513, a place of employment 514, an e-mail address,a telephone number, and so forth. If the surveillance video brokerservices are provided for a fee, client table may further containbilling and payment information, although such information may becontained in a separate table (not shown).

A clients table entry 410 may further contain an emergency authorizationflag 515, indicating whether the corresponding client has authority todeclare a state of emergency. As explained herein, in one or moreembodiments, an emergency condition may be temporarily declared by anauthorized client, such as an appropriate public official, which can beused to facilitate greater access to surveillance video than mightotherwise be possible. Clearly, such authority is subject to potentialabuse, and therefore should be carefully limited. The emergencyauthorization flag 515, if present, is not directly accessible to oreditable by the client, but can only be altered by a systemadministrator or similar individual who manages broker 103.Alternatively, emergency authorization is not stored in clients table410 at all, but is stored in a separate record for security purposes(although it may logically be considered part of the client record).

Client groups table 414 contains multiple records, each recordscorresponding to a respective client group. In one aspect of thepreferred embodiment, each client may belong to one or more clientgroups for purposes of obtaining services from broker 103. A group maybe defined by a client, a surveillance domain, or by the video brokeritself. In particular, the use of client groups allows a surveillancedomain to grant video access to a group, without having to determine theidentity of each member of the group. For example, access may be grantedto a local fire department as a group, without the surveillance domainowner having to determine the identities of each employee of the localfire department, and without having to maintain such a list as employeesenter or leave the fire department. FIG. 6 is a conceptual illustrationshowing in greater detail a client groups table 414 in a video brokerdatabase 404, according to the preferred embodiment. Each record 601 ingroups table 414 contains a group identifier (ID) 602 and anadministrator identifier (ID) identifying a client (or another group)having administrative authority for the group, i.e., having authority todelete from or add members to the group, and could optionally containadditional information about the group.

Group membership table 413 is used in conjunction with groups table 414to define the members of the group. FIG. 7 is a conceptual illustrationshowing in greater detail a group membership table 413 in a video brokerdatabase 404, according to the preferred embodiment. Each record 701 ingroup membership table 413 corresponds to a single instance of groupmembership, i.e., it contains a client ID 702 identifying a client and agroup ID 703 identifying a group to which the client belongs. It will beobserved that a client may belong to multiple groups by having multiplerecords in the group membership table.

Surveillance domains table 412 contains multiple records, each recordcorresponding to a respective surveillance domain 102 in networkedsurveillance system environment 100. FIG. 8 is a conceptual illustrationshowing in greater detail a surveillance domains table 412 in a videobroker database 404, according to the preferred embodiment. Each record801 in surveillance domain table 412 contains a surveillance domainidentifier (ID) 802 and a password 803. A surveillance domain record 801and may contain any of various additional fields useful to define theparameters of the surveillance domain. For example, where some of thedomains are mobile domains, surveillance domain record 801 preferablycontains a mobile flag 804 indicating whether the domain is a mobiledomain. A privacy flag 805 may be used to indicate whether thesurveillance domain is visible to clients not authorized to access it.I.e., if a client performs a search of surveillance domains, asdescribed further herein, privacy flag will prevent the surveillancedomain from being disclosed to the client in the search results if theclient is not authorized to access surveillance video of the domain.Further information in record 801 may include any or all of: a locationof a surveillance domain 806 (this information being periodicallyupdated in the case of a mobile domain); the bandwidth of networkconnection(s) available to the domain 807; protocol used by the videosignal output by the domain 808; a video access location specifyingwhere or how the surveillance video may be accessed, such as a networkaddress of a video distribution service 105 or a network address of aserver within the surveillance domain which provides the video; thenumber of cameras in the domain 810, a name 811, email 812 and phonenumber 813 of a responsible contact person, a natural languagedescription of the domain 814 (e.g. “Carl's Sporting Goods store”), astreet address of the domain 815, and such other data as may be usefulto describe the domain or its characteristics.

Cameras table 411 contains multiple records, each record correspondingto a respective video surveillance camera 201. FIG. 9 is a conceptualillustration showing in greater detail a cameras table 411 in a videobroker database 404, according to the preferred embodiment. Each record901 in cameras table 411 contains a camera ID 902 of the correspondingcamera, a surveillance domain ID 903 identifying the surveillance domainto which the camera belongs (each camera belonging to one and only onesurveillance domain in the preferred embodiment), and a camera group ID904 identifying an optional camera group within the surveillance domain.Both the camera ID and the optional camera group ID may be assigned bythe surveillance domain, and need only be unique within thatsurveillance domain. The purpose of the camera group is to allowmultiple cameras to be designated by a single authorization. Its purposeis therefor similar to the client group, but because a single camera canbelong to only a single group, and that group must be part of asurveillance domain, there is no separate camera groups table or cameragroup membership table, as there is in the case of client groups. Itwould alternatively be possible to allow each camera to be a member ofmultiple camera groups by using additional tables as in the case ofclient groups. A record 901 may contain any of various additional fieldsuseful to define the parameters of the camera, in particular a locationof the camera 905 and azimuth of its orientation 906, any of variouscontrol parameters 907 which may be available, such as zoom ordirectional controls, a natural language description 908 of the camera,in particular its field of view (e.g., “front door”, “parking lot”,etc.), and any other useful data. The location of the camera may beunnecessary for many surveillance domains, given that the domain itselfhas a location, but there are some relatively large domains for whichlocation information could be important.

Notifications table 408 and filters table 409 are used to regulate thetransmission of alert notifications generated in a surveillance domainto one or more clients, notifications table 408 being used to specifyauthorized recipients of alert notifications and filters table 409 beingused to specify filters parameters of recipients. Notifications table408 contains multiple records, each record corresponding to a pair of asurveillance domain producing an alert and a client (or client group)authorized to receive the alert notification. Authorization to receivethe notification is granted in the first instance by the surveillancedomain. In the preferred embodiment, the surveillance domain can specifyunconditional authorization of a client to receive notifications, orauthorization conditioned on one or more events. These events could bescheduled events (e.g., a time of day, day or week and/or range ofdates), or unscheduled events (e.g., a fire, an intrusion into amonitored area, etc.). For example, a building owner or tenant of anarea covered by a surveillance domain may wish to see all alertsgenerated by the domain (unconditional notification), while notificationto selective others is restricted to particular times and/or conditions(for example, a police department receiving alert notification or aperimeter intrusion only at times when nobody should be on thepremises), and notification to the general public is not provided atall.

FIG. 10 is a conceptual illustration showing in greater detail an alertnotifications table 408 in a video broker database 404, according to thepreferred embodiment. Each record 1001 in notifications table 408contains a corresponding surveillance domain ID 1002 of the sourcesurveillance domain of the alert; a recipient ID 1003 of the of theclient or client group to which notification is to be sent; an alerttype 1004 of the applicable alert; a start date/time 1005 and enddate/time 1006 of a time window within which the alert must be received;a set of day of week flags 1007 for alert time windows which recur on aweekly basis, and a daily start time 1008 and daily end time 1009 forthe time windows which recur on a weekly basis; and such additionalconditions as may be deemed desirable. In any of these fields (exceptthe surveillance domain ID), a null or other appropriate value can beused to designate a “wild card”, meaning that any value of thecorresponding field will satisfy the filter parameters. The parametersspecified by the field values are logically ANDed, so that allparameters must be satisfied in order for the alert notification to beauthorized for the recipient client or client group. Multiple records onbehalf of a single requestor may be placed in notifications table 408 toaddress logical OR conditions.

Alert filters table 409 contains multiple records, each recordcorresponding to a set of parameters of a requested alert notificationto be provided by broker 103. In the preferred embodiment, clients havethe ability to filter the alerts for which they will receivenotification. This is of particular benefit to large, public entities,such as a local police department, which might otherwise be inundatedwith automatically generated alert notifications. It might further beused to direct alerts to a specific individual within a largeorganization. For example, a surveillance domain may authorize thesending of alerts to a client group (such as a police department), notknowing the identity of the individual to handle alerts. The policedepartment could then use the filter to prevent alert notifications frombeing sent to each and every member of the department, and to permit thenotification to reach a single individual (or very select number ofindividuals). A client (or a client acting on behalf of a client group)therefore specifies filter parameters for alerts it is to receive, eachrecord in filters table 409 corresponding to a single client (or aclient group) and types of alerts to be received.

FIG. 11 is a conceptual illustration showing in greater detail an alertfilters table 409 in a video broker database 404, according to thepreferred embodiment. Each record 1101 in filters table 409 contains acorresponding recipient ID 1102 of the client or group to which thenotification parameters applies; a source ID 1103 specifying a sourcesurveillance domain to which the filter applies (i.e., the filter recordapplies to alerts generated by the corresponding source surveillancedomain); an alert type 1104 of the applicable alert; a start date/time1105 and end date/time 1106 of a time window within which the alert mustbe received; a set of day of week flags 1107 for alert time windowswhich recur on a weekly basis, and a daily start time 1108 and daily endtime 1109 for the time windows which recur on a weekly basis; an alertsource location area 1110; specifying a location within which the alertmust be generated; and such additional conditions as may be deemeddesirable. The parameters specified by the field values are logicallyANDed, (“wild card” values being possible where appropriate), so thatall parameters must be satisfied in order for the alert notification tobe sent to the requestor. Multiple records on behalf of a singlerequestor may be placed in filters table 409 to address logical ORconditions.

Video authorizations table 406 contains records of outstandingconditional and unconditional authorizations to access video producedfrom surveillance domains 102. Each record in authorizations table 406corresponds to a pair of a surveillance domain producing video data anda client (or client group) authorized to receive the video data, theauthorization to receive the video data being granted in the firstinstance by the surveillance domain. The surveillance domain can specifyunconditional authorization of a client to receive video data, orauthorization conditioned on one or more events. As in the case of alertnotifications, these events could be scheduled events (e.g., a time ofday, day or week and/or range of dates), or unscheduled events (e.g., afire, an intrusion into a monitored area, etc.).

FIG. 12 is a conceptual illustration showing in greater detail a videoauthorizations table 406 in a video broker database 404, according tothe preferred embodiment. Each record 1201 in authorizations table 406contains a corresponding domain ID 1202 of the source surveillancedomain of the video to be authorized; a camera ID 1203 containing theidentifier of a camera or camera group within the source surveillancedomain to which the authorization is limited; a recipient ID 1204 of theof the client or client group authorized to receive the video; andmultiple fields specifying conditions, if any, under which the recipientis authorized to receive the video. It is expected that mostauthorizations will grant authority to view video from any camera withinthe surveillance domain, in which case an appropriate “wild card” valuewill be used in camera ID field 1203. However, there may becircumstances in which it is desirable to grant access to video from asingle camera or group of cameras within the domain. For example, alarge business may have areas of varying sensitivity, from outdoorsareas such as grounds and parking lot which are visible to the generalpublic at all times, to semi-public areas such as lobby, to ordinarywork areas such as offices, laboratory, or manufacturing area which arenot normally open to the public, to areas of extreme sensitivity whereimportant company matters and secrets may be kept or discussed. In suchcases, it may be desirable to grant access to different cameras withinthe surveillance domain under different circumstances.

The conditions for granting video access authority could include any ofvarious scheduled or non-scheduled events, the conditions fieldsspecifying, e.g., one or more sensor-generated alert events; a startdate and end date of a time window within which video access isauthorized; a start time, end time and day(s) of week of time periodswhich are scheduled to repeat on a weekly basis, during which access isauthorized; a client location area condition requiring that thereceiving client be within a specified area; and/or other conditions. Inthe exemplary video authorizations table of FIG. 12, fields 1205-1209specify scheduled events, i.e., a start date/time 1205 and end date/time1206 of a time window within which video access is authorized; a set ofday of week flags 1207 for access authorization time windows which recuron a weekly basis, and a daily start time 1208 and daily end time 1209for the time windows which recur on a weekly basis. Fields 1210-1213specify non-scheduled event conditions which may cause access to beauthorized. Unconditional flag 1210 is set to indicate that there are nonon-scheduled event conditions on the granting of access, i.e., accessis granted regardless of the state of event alerts (although it maystill be restricted to scheduled times). If unconditional flag 1210 isnot set, then condition flags C1 . . . Cn 1211 indicate which conditionswill cause access to be authorized. The conditions corresponding to thevarious flags C1 . . . Cn could be any of the conditions previouslymentioned, i.e, sensor generated alarms, location proximity of a client,etc. One of the conditions is a locational proximity condition, in whichthe client is granted access if within a certain locational proximity(preferably expressed as a distance) from the surveillance domain.Proximity field 1212 expresses the required proximity for grantingaccess. Condition data may include one or more duration fields 1213 (ofwhich one is shown) for access authorizations granted as a result of anon-scheduled event, i.e., the duration field specifies that access isavailable beginning with the time of the non-scheduled event and endingafter the time duration specified in the duration field has elapsed.Although not shown in FIG. 12, there could be separate duration fieldsfor each type of event. The parameters specified by the field values arelogically ANDed (“wild card” values being possible where appropriate),so that all parameters must be satisfied in order for video access to beauthorized for the recipient client or client group. Multiple recordsmay be used in authorizations table 406 to address logical ORconditions.

In particular, in accordance with one or more preferred embodiments, atleast one condition specified by condition flags C1 . . . Cn 1210corresponds to a “declared emergency” condition. A “declared emergency”condition is an emergency declared by an appropriate public authority,such as a police department, fire department, emergency response agency,and so forth. An owner or other person in control of a privatesurveillance domain can pre-authorize persons to access surveillancevideo upon the declaration of such an emergency, without the need toobtain permission when time may be critical, by setting the appropriatecondition flag for the “declared emergency” condition. There could bemultiple different types of declared emergency which are independentlyset to grant video authorization.

While it would be possible to grant authorization to receive video datato exactly the same clients or client groups and under exactly the sameconditions as authorization to receive alert notifications (and thus usea single table of authorizations rather than a separate videoauthorizations and alert notifications table), in the preferredembodiment the two forms of authorization are granted independently.There are various instances in which the capability to independentlygrant authorization to view video and authorization to receive alertsmight be useful. For example, if the camera is in an area which is opento the general public (a parking lot, a courtyard, etc.) theauthorization to access the video might be very broad (even to includethe general public), but it would be undesirable to send alertnotifications to such a broad set of clients. As another example, insome instances privacy concerns may be so strong that the access rightsto the video are extremely limited even when an alert notification isprovided. For example, an alert concerning an unauthorized intruder in ahighly sensitive area might be sent to a police department (i.e., aclient group), but authorization to the video might be limited to thoseofficers who actually respond, in this case providing the video to aportable client device only when the location of the responding officersis within some sufficiently close distance of the site.

Events table 405 contains records of non-scheduled events which haveoccurred and which may be the basis for sending alert notificationsand/or authorizing access to video, each record in events table 405corresponding to a single detected event. FIG. 13 is a conceptualillustration showing in greater detail an events table 405 in a videobroker database 404, according to the preferred embodiment. Each record1301 in events table 405 contains a source ID 1302 specifying the sourceof the event, i.e. the surveillance domain ID or client ID of thesurveillance domain or client, as the case may be, which originated theevent; an event type 1303; a date and time of the event 1304; an“active” flag 1305 indicating whether the event is still considered“active”, i.e. access may be granted based on the event; and such otherinformation as may be useful to characterize the event. Preferably,events are manually deactivated by the originator, and thereafter cannot serve as a basis for granting access to video. Deactivated eventsmay be stored in a separate partition of events table to reduce the sizeof the table when searching for active events.

Video accesses table 407 is intended as a log of video accessauthorizations which have been granted by broker 103. FIG. 14 is aconceptual illustration showing in greater detail a video accesses table407 in a video broker database 404, according to the preferredembodiment. Each record 1401 in video accesses table 407 records asingle instance of video access authorization granted to a singleclient, and contains a surveillance domain ID 1402 to which access wasgranted; a camera ID 1403 of a camera (or camera group) to which accesswas granted; a client ID 1404 of the client granted access; a date andtime the access was granted 1405; a date and time the access ended 1406;and such other information with respect to the access as may be useful.Although an entire client group may be authorized to obtain access toselective video, each individual client granted access is recorded as aseparate entry in accesses table 407. Accesses table 407 provides arecord, which can be searched using standard database search techniques,of all past accesses, and is important as a tracking and auditingvehicle to ensure that accesses are appropriately granted. It is notedthat in one or more embodiments herein, a party other than the owner ofthe surveillance domain may grant access. In particular, a public agencyemergency services provider (fire, police, medical emergency responder,etc.) may declare an emergency and thereby obtain access to video,without pre-authorization of the surveillance domain owner. The primarysafeguard against abuse of this capability is the audit trail providedby accesses table 407.

In the above descriptions, a location or location area is preferablyspecified as a set of global positioning coordinates corresponding to alocation (if the location is intended to be a specific point, such asthe location of a camera), and as a set of global positioningcoordinates corresponding to a location center and a radius from thecenter (if the location is intended to be an area or region which ismatched to a point, such as the location of an area or region specifiedin a filter). It will be appreciated that other techniques could be usedto specify a location, such as quadrants or grid locations on a citymap. Additionally, location may be provided in multiple formats, such asa set of global positioning coordinates which is used for automaticmatching of location information in the database, and an accompanyingnatural language description (such as a street address, floor, wing orroom of a building) which is more readily understandable to a humanuser. If the location applies to a mobile device (such as a mobilesurveillance domain or mobile client), the location may be periodicallyupdated to provide reasonably current location data.

Video broker database manager 420 is executable computer programmingcode which executes on CPU 301 to provide basic functions for themanagement of database 403. Database manager 420 may theoreticallysupport an arbitrary number of database tables, which may or may nothave related information, a specific number and structure of tablesbeing shown in FIG. 4. Database manager 420 preferably containsadministrative functions 421 allowing users to perform basicadministrative operations with respect to the database, such as definingand editing database table definitions, creating, editing and removingrecords in the database, viewing records in the database, definingdatabase auxiliary data structures such as indexes and materializedquery tables, and so forth. In general, it is expected that theseadministrative functions would be performed by employees or contractorsof the broker 103. Database manager 420 preferably further includes aquery engine 422 for executing queries against data in database tables405-414. Database management system further preferably includes anexternal interface 423 having one or more application programminginterfaces (APIs) by which external applications can access data invideo broker data 404 either by invoking query engine 422 or throughother means. Database management system 420 may further contain any ofvarious more advanced database functions, as are known in the art. Inthe preferred embodiment, database manager 420 is a generic databasemanagement system implementing a structured query language (SQL) queryprotocol, it being understood that a database manager mightalternatively query and structure data according to some other protocoland/or that it might be a custom designed database management system.Although database manager 420 is represented in FIG. 4 as an entityseparate from operating system kernel 401, it will be understood that insome computer architectures various database management functions areintegrated with the operating system.

Although one database 403 having ten database tables 405-414 and threeauxiliary structures 415-417 are shown in FIG. 4, the number of suchentities may vary, and could be much larger. Additionally, database 403may be logically part of a larger distributed database which is storedon multiple computer systems. Although database manager 420 isrepresented in FIG. 4 as part of database 403, the database manager,being executable code, is sometimes considered an entity separate fromthe “database”, i.e., the data.

Memory 302 of a computer system 300 configured as a video broker 103further includes video broker controller 430. Video broker controller430 is a set of computer programs or functions (i.e., executablecomputer programming code which execute on CPU 301) providing basicfunctions for the operation of broker and accessing of data in database403 needed to perform broker operations. In particular, the video brokercontroller includes a registration function 431, a surveillance domain(SD) service function 432 which includes an alert notifications handler433, and a client service function 434 which includes a videoauthorization function 435.

Registration function 431 is used for registering clients andsurveillance domains to video broker 103, and may also be used forinitial input of relevant profile data defining the clients andsurveillance domains and stored in various files of database 403.

SD Service function 432 provides service to registered surveillancedomains, which in particular includes editing of profile andauthorization information such as stored in any of surveillance domainstable 412, cameras table 411, alert notifications table 412, and/orvideo authorizations table 414, and the receipt and handling of alertnotifications. Alert notifications handler 433 handles the generationand transmission of appropriate alert notifications responsive todetected events. Alert notifications handler accesses appropriaterecords in database 403 to determine the identity of clients to receivenotification of each event, determines the appropriate filteringparameters and method of notification, and generates and transmits oneor more alert notifications as required to each client entitled tonotification.

Client Service function 434 provides service to registered clientdevices which access broker 103. This service includes: editing ofprofile, group and filter information stored in any of clients table410, client groups table 414, group membership table 413 and/or filterstable 413; search and discovery of surveillance domains usinginformation in database 403; declaration of emergency state; searchingthe log of video accesses in accesses table 407; and granting access tosurveillance video in accordance with previously specified accessparameters. Video access function 435 handles the grant of access tocertain video responsive to requests from clients. Video access functionreceives each client request to access designated video, accessesappropriate records in database 403 to verify authority to access therequested video, determines whether access is authorized, and if sogrants access rights to the requesting client and logs the access.

In addition to database management system 410 and video brokercontroller 430, one or more user applications (not shown) may accessdata in database 403 to perform tasks on behalf of one or more users,and may use defined auxiliary database structures to do so. Such userapplications may execute on computer system 300, or may access thedatabase from remote systems. Such user applications may include, e.g.,accounting applications which allocate charges for services provided;audit and security applications which monitor events and accesses;administrative applications which maintain the video broker system; orany of various other applications which may have use for the data.

Various software entities are represented in FIG. 4 as being separateentities or contained within other entities. However, it will beunderstood that this representation is for illustrative purposes only,and that particular modules or data entities could be separate entities,or part of a common module or package of modules. Furthermore, althougha certain number and type of software entities are shown in theconceptual representation of FIG. 4, it will be understood that theactual number of such entities may vary, and in particular, that in acomplex database environment, the number and complexity of such entitiesis typically much larger. Additionally, although software components403-417, 420-423, and 430-435 are depicted in FIG. 4 on a singlecomputer system 300 for completeness of the representation, it is notnecessarily true that all programs, functions and data will be presenton a single computer system or will be performed on a single computersystem. In particular, video broker controller 430 and/or associateddata or functions could be located on a system separate from brokerdatabase 403, the data in broker database 403 being accessed remotely byfunctions within video broker controller 430.

While the software components of FIG. 4 are shown conceptually asresiding in memory 302, it will be understood that in general the memoryof a computer system will be too small to hold all programs and datasimultaneously, and that information is typically stored in data storagedevices 325-327, comprising one or more mass storage devices such asrotating magnetic disk drives, and that the information is paged intomemory by the operating system kernel 401 as required. In particular,database tables 405-414 are typically much too large to be loaded intomemory, and typically only a small portion of the total number ofdatabase records is loaded into memory at any one time. The fulldatabase 403 is typically recorded in disk storage 325-327. Furthermore,it will be understood that the conceptual representation of FIG. 4 isnot meant to imply any particular memory organizational model, and thatsystem 300 might employ a single address space virtual memory, or mightemploy multiple virtual address spaces which overlap.

In accordance with the preferred embodiment, surveillance video broker103 connected to network 101 serves as a clearinghouse for accessingvideo surveillance data produced by surveillance domains 102 by clients104 and forwarding alerts to clients 104. Both surveillance domains andclients independently register with surveillance video broker 103 foruse of its services, providing such information as may be required. Onceregistered, surveillance domains may thereafter send alert notificationsto the surveillance video broker for transmission to one or moreclients. Clients may access the surveillance video broker for any ofvarious services, but in particular to gain access to surveillance videoproduce by a surveillance domain.

Registration may use any of various known techniques for verifying theidentity of the registering surveillance domain or client, and assigningsecurity protection mechanisms to the relationship. Part of theregistration process may be manually-assisted, but at some pointessential registration information is entered into surveillance videobroker 103 and stored in its database 403. For example, an on-lineregistration request providing essential information could be verifiedby mail confirmation to a street address or in-person confirmation.Preferably, at a minimum, a registering surveillance domain or clientwould be assigned a user identifier and password for accessing broker103 over network 101. Additional security mechanisms could be providedfor access and/or protection of information, such as challengequestions, encryption keys for encrypting sensitive data, etc.

When a surveillance domain registers with the surveillance video broker103, it provides the information stored in the corresponding entry 801for that surveillance domain in surveillance domains table 412. Thisinformation is described above with respect to FIG. 8. The surveillancedomain further provides information for a corresponding entry 901 incameras table 411 for each camera within the surveillance domain. Thisinformation is described above with respect to FIG. 9. The surveillancedomain further optionally provides information for one or morenotification requests, each request being stored in a correspondingentry 1001 in notifications table 408, and directing broker 103 tonotify a respective client or client group upon the occurrence ofspecified events. This information is described above with respect toFIG. 10. The surveillance domain further optionally provides informationfor one or more video access authorizations, each video accessauthorization being stored in a corresponding entry 1201 in videoauthorizations table 406, and directing broker 103 to grant video accessto a respective client or client group under specified conditions. Thisinformation is described above with respect to FIG. 12.

Similarly, a client registers with the surveillance video broker 103 andprovides the information stored in the corresponding entry 501 for thatclient in clients table 410. This information is described above withrespect to FIG. 5. The client further optionally provides informationfor one or more notification filter entries 1101 in filters table 409,each filter entry specifying conditions under which an applicable alertnotification is to be transmitted to the client. This information isdescribed above with respect to FIG. 11.

The relevant information may be provided by the surveillance domain orclient at the time of registration or thereafter. Typically,notification requests, notification filters, and video accessauthorizations are updated from time to time, and therefore thesurveillance domain or client is able to log into broker 103 and alterpreviously submitted data or provide new data. Thus, in the case ofvideo access authorizations or notification requests, the surveillancedomain maintains complete control of the outstanding authorizations toaccess its video as well as the notification requests, and can changethese at any time. The client (or client group) maintains completecontrol of its notification filters. The data may be provided by manualentry (e.g., by entering data in an on-line interactive screen), orautomatically by use of a computer program which gathers the data anduploads it to the broker, or by a combination of manual and automatedmeans. For example, broker 103 may provide downloadable computerprograms for certain popular operating systems which gatherconfiguration data from the surveillance domain or client and upload it.Data such as global positioning coordinates, video protocols andbandwidth, camera control data, and so forth, may be automaticallyaccessible and readily gathered by an appropriate automated program, toreduce possible error in manual entry. Even where automated programsexist to upload certain configuration data, it is likely that videoaccess authorizations, notification requests and notification filterswill be manually specified.

Once registered, the surveillance domain remotely logs in to broker 103to utilize its services. The surveillance domain is logged in to changeits profile data, change the alert notifications or video accessauthorizations, or to transmit an alert to the broker. Preferably, thesurveillance domain is continuously logged in for relatively longperiods of time while the services of broker 103 are utilized to sendnotifications or facilitate access to surveillance video, but dependingon the implementation, it could alternatively log in only intermittentlywhen it desires service from the broker. The mechanism for surveillancevideo access may, but need not necessarily, require that thesurveillance domain be logged in to the broker in order for a client toaccess its video, as discussed in greater detail herein. Where thesurveillance domain is required to be logged in for video access and login is intermittent, the broker will send a message to the surveillancedomain using any appropriate mechanism requesting it to log in in theevent that a client requests access to the surveillance domain's video.

FIG. 15 is a flow diagram illustrating at a high level the operation ofSD service function 432 executing in surveillance video broker 103, andproviding service to a requesting surveillance domain 102 connected overnetwork 101, according to the preferred embodiment. Referring to FIG.15, a previously registered surveillance domain 102 issues a loginrequest, which is received by broker 103 (block 1501). The brokerresponds by soliciting the necessary data to verify the identity of thesurveillance domain, and using the data provided to verify the domain'sidentity (block 1502). Identity of the surveillance domain is preferablyverified at login time by soliciting a user ID and password from thesurveillance domain. For greater security, other and/or additional formsof identity verification could be used, for example, challenge questionscould be asked, or randomly generated challenge data could be sent forencryption or decryption by the surveillance domain using a key of apublic/private key pair, as is known in the art. The data provided bythe surveillance domain might be manually entered or automaticallyprovided (e.g., from stored data). In particular, where the surveillancedomain logs in only intermittently, login information would typically beautomatically provided to facilitate the automatic transmission of alertnotifications.

Once the identity of the surveillance domain is verified, the domain islogged on to the broker and the broker waits for an incoming servicerequest from the surveillance domain, represented as the tight loop atblock 1503. There could be a very considerable time delay between log inand the submission of a request. The representation of a delay loop atblock 1503 is not meant to imply that the SD service function 432 iscontinually executing during this period, but as is well known, the SDservice function may be placed in an idle state by OS kernel 401 whileawaiting an incoming request to allow other processes to execute inprocessor 301. Upon receipt of a request, the request is processed,represented as the ‘Y’ path from block 1503.

If the request is a request to transmit an alert notification, the ‘Y’branch is taken from block 1504, and the alert notification is processedby querying the database to determine which clients should receivenotification, and generating and transmitting the appropriate alertnotifications to clients (represented in FIG. 15 as block 1505, andshown in greater detail in FIG. 16). After servicing the request,control returns to block 1503 to await the next request. In thepreferred embodiment, an alert notification is a form of service requestfrom the surveillance domain to the broker, and therefore thesurveillance domain is logged in to the broker in order to make therequest, thus assuring that the surveillance domain's identity has beenverified and the possibility of false alerts minimized. It wouldalternatively be possible to provide alert notifications to the brokerby other means which do not require login, such as instant messaging,e-mail or telephonic transmission, but in those cases some alternativeprovision should be made for verifying the identity of the source of thealert.

FIG. 16 is a flow diagram illustrating in greater detail a process forhandling an alert notification request, according to the preferredembodiment. In the representation of FIG. 16, actions performed by asurveillance domain 102 requesting alert notification are shown in theleft column, while actions performed by surveillance video broker 103are shown in the middle column, and actions performed by a clientreceiving a notification are shown in the right column.

Referring to FIG. 16, a previously registered surveillance domain 102detects an unscheduled alert event for which there is cause to send analert notification (block 1601). The alert event might be any of variousevents detected by any of sensors 204, including sensors not representedin any of FIGS. 2A, 2B, or described in connection therewith. By way ofillustration and not limitation, an alert event may include any or allof: a perimeter intrusion detected by a perimeter sensor; motion withina defined area detected by a motion detector; a potential fire detectedby a smoke and/or heat sensor, intrusion in an area detected by anauditory sensor, glass breakage (e.g. window) detected by auditory,contact film or other glass breakage sensors; extreme high or lowtemperature conditions detected by temperature sensors; gas leakagedetected by sensors which detect a gas presence or leakage from anenclosed vessel, such as a pressure sensor; water leakage or penetrationdetected by appropriate sensors, chemical spill detected by appropriatechemical, pressure or weight sensors, excessive radiation detected byradiation sensors; and so forth. An alert event might further includeany of various manually-tripped alarms, such as well-known manuallyoperated fire alarms, bank teller or store clerk alarms, and the like.An alert event might further include any malfunction of surveillancedomain equipment, which may be an indication of unauthorized tampering,or simply an item requiring maintenance. Where the surveillance domainis a mobile surveillance domain, an alert event might further includeany or all of: the domain equipment (e.g. a motor vehicle) venturingoutside a pre-defined area, as detected by a global positioning device;the domain equipment exceeding a pre-determined speed; the domainequipment experiencing a collision, etc. An alert event might further bea condition which is detected from video analysis of the surveillancevideo itself. For example, analysis may detect the presence of anintruder in an area under surveillance or some other condition requiringand alert.

If the surveillance domain 102 is not logged in to broker 103 when thealert condition is detected (e.g., login is intermittent rather thancontinuous), the surveillance domain logs in to broker 103, generallyrequiring an exchange of login data for verification purposes. This isrepresented in FIG. 16 as optional blocks 1602 and 1603. Preferably, thesurveillance domain stores the necessary login data and automaticallylogs in to the broker, without requiring manual data entry from anoperator.

The surveillance domain then transmits an alert notification request tobroker 103 over network 101 (block 1604). The alert notification requestincludes any necessary parameters of the alert, e.g., an alert type, alocation or other identification of a sensor which detected thecondition, and so forth. The alert notification is received by thebroker (block 1605).

Upon receipt of the request, alert notifications handler 433 executingin broker 103 queries database 403 to determine those clients who areauthorized to receive notification of the alert (block 1606). A clientauthorized to receive alert notification is one who is either (a)identified as a recipient, or (b) a member of a client group identifiedas a recipient, in the recipient ID field 1003 of an entry 1001 in alertnotifications table 408 which matches the parameters of the receivedalert notification request. This can be determined using standarddatabase query techniques. For example, a query of the form:

FIND clientID in ClientsTable    WHERE clientID =NotificationsTable.recptID AND       [domainID of requestor] =NotificationsTable.domainID AND       [type of alert] matchesNotificationsTable.type AND       [other alert parameters] matchNotificationsTable.[other       parameters]would return clients directly identified as recipients, and a query ofthe form:

FIND clientID in ClientsTable    WHERE clientID = GrpMembTable.clientIDAND       GrpMembTable.groupID = NotificationsTable.recptID AND      [domainID of requestor] = NotificationsTable.domainID AND      [type of alert] matches NotificationsTable.type AND       [otheralert parameters] match NotificationsTable.[other       parameters]would return clients who are members of a group identified as arecipient.

The alert notifications handler 433 further filters the authorizedclients as determined above to determine those authorized clients whoactually wish to receive notification (block 1607). Again, this can bedone using standard database query techniques. For example, a query ofthe form:

FIND clientID in [results from block 1606]    WHERE clientID =FiltersTable.recptID AND       [domainID of requestor] matchesFiltersTable.domainID       [type of alert] matches FiltersTable.type      [other alert parameters] match FiltersTable.[other parameters]would return clients directly identified as recipients recipientID field1102 of a corresponding entry 1101 in filters table 409, and a query ofthe form:

FIND clientID in [results from block 1606]    WHERE clientID =GrpMembTable.clientID AND       GrpMembTable.groupID =FiltersTable.recptID AND       [domainID of requestor] matchesFiltersTable.domainID       [type of alert] matches FiltersTable.type      [other alert parameters] match FiltersTable.[other parameters]would return clients who are members of a group identified as arecipient.

The resulting list of clients to receive notification is pruned ifnecessary to remove any duplicates (block 1608). While blocks 1606-1608have been illustrated and described herein as separate, sequentialoperations, it will be appreciated by those skilled in the art ofdatabase design and queries that various query conditions expressedherein might be combined or performed in different order for purposes ofefficiency, and that database tables and or auxiliary structures mightbe partitioned, indexed or otherwise constructed to efficiently supportextraction of the information explained herein.

Having determined the clients to receive notification of the alert,alert notifications handler 432 generates the appropriate alertnotification(s) to each client and causes them to be transmitted (block1609). Alert notification may be transmitted by any of various means,including e-mail, voice message, text message and/or instant message, aspreviously specified for each client in the client's record 501 inclients table 410. A client may specify multiple forms of notificationto increase the probability of timely receipt.

The alert notifications handler 432 further generates an acknowledgmentmessage back to the requesting surveillance domain and causes themessage to be transmitted to the surveillance domain (block 1610). Thesurveillance domain receives and stores the acknowledgment message(block 1611). Generally, no further action is required by thesurveillance domain. If the surveillance domain is not continuouslylogged in to the broker, the surveillance domain will then log off(blocks 1612 and 1613).

On the client side, a client device may receive a notification (block1614), and take appropriate action in response (block 1615). Generally,appropriate action will be to bring it to the attention of a human user,as by display of alert data or selectable icon representing the alert,an audible sound indicating the presence of an incoming message, and soforth. Typically, the human user will then make a determination as toappropriate further action. However, a client device may in appropriatesituations also take some automatic action. For example, a client devicecould automatically request authorization to surveillance videoresponsive to an alert and display the surveillance video to a user,thus removing the need for the user to manually request authorization inorder to further investigate the alert.

Referring again to FIG. 15, if the incoming request from thesurveillance domain is something other than an alert notificationrequest, the ‘N’ branch from block 1504 is taken.

A surveillance domain may request to add, delete or otherwise editauthorizations to access surveillance video produced by it, representedas the ‘Y’ branch from block 1506. In this case, SD service function 432presents one or more interactive selection and editing screens to a userin the surveillance domain allowing the user to search for a previousauthorization (stored in video authorizations table 406), edit or deletea previous authorization when found, and/or generate a new authorization(block 1507). This may be accomplished, e.g., by presenting one or moreinteractive web pages to a user executing a browser application withinsurveillance domain local controller 202. Any changes are stored in oneor more appropriate entries in video authorizations table 406, and theSD service function returns to the idle loop at block 1503.

If the incoming request is neither an alert notification request nor arequest to edit access authorizations (the ‘N’ branch from block 1506),a surveillance domain may further request to add, delete or otherwiseedit alert notification records for alerts generated by it, representedas the ‘Y’ branch from block 1508. In this case, the SD service functionpresents one or more interactive selection and editing screens to asurveillance domain user allowing the user to search for a previousnotification record (stored in notifications table 408), edit or deletea previous notification record when found, and/or generate a newnotification record (block 1509). This may be accomplished, e.g., bypresenting one or more interactive web pages to a user executing abrowser application within surveillance domain local controller 202. Anychanges are stored in one or more appropriate entries in notificationstable 408, and the SD service function returns to the idle loop at block1503.

If the incoming request is neither an alert notification request nor arequest to edit access authorizations nor a request to edit notificationrecords (the ‘N’ branch from block 1508), a surveillance domain mayfurther request to edit other profile data, such as data defining theparameters of the surveillance domain or its cameras, represented as the‘Y’ branch from block 1510. In this case, the SD service function maypresent one or more interactive selection and editing screens to asurveillance domain user allowing the user to edit the desiredinformation, e.g., by presenting interactive web pages to a userexecuting a browser application within surveillance domain localcontroller 202 (block 1511). Alternatively, block 1511 may representcertain information provided or edited by an automated program executingin a surveillance domain. For example, in the case of a mobilesurveillance domain, it may be desirable to periodically update thelocation of the surveillance domain. It would be inconvenient and errorprone for a user to do this manually, but a surveillance domain localcontroller could be programmed to automatically provide such informationto the broker via a series of requests, either at pre-determinedintervals or when a current location becomes sufficiently distant from apreviously supplied location. Any changes are stored in appropriatelocation(s) in database 403, and the SD service function returns to theidle loop at block 1503.

If the incoming request is neither an alert notification request nor arequest to edit access authorizations nor a request to edit notificationrecords nor a request to edit other profile data (the ‘N’ branch fromblock 1510), a surveillance domain may further request other services(not including logout), represented as the ‘N’ branch from block 1512.In this case, the SD service function handles the request as appropriate(block 1513). Other request may include, e.g., requests for maintenanceand diagnostic services, accounting and billing matters, or any otherservice which might be useful. After servicing the request, the SDservice function returns to the idle loop at block 1503. If the requestis a logout request (the ‘Y’ branch from block 1512), the SD servicefunction terminates and the surveillance domain is logged out.

Client services function 434 executing in broker 103 similarly handlesrequests for service from clients 104. Once a client is registered, theclient remotely logs in to broker 103 to utilize its services. Servicesprovided by the broker to the client include: obtaining video accessauthorization to access surveillance video produced by a surveillancedomain; searching for available surveillance domains; changingnotification filters parameters, or other client profile data; definingmembership in a client group; declaring an emergency condition to enablecontrolled video access; and/or examining records of past events orvideo accesses.

FIGS. 17A and 17B (herein collectively referred to as FIG. 17) are aflow diagram illustrating at a high level the operation of clientservice function 434 executing in surveillance video broker 103, andproviding service to a requesting client 104 connected over network 101,according to the preferred embodiment. Referring to FIG. 17, apreviously registered client 104 issues a login request, which isreceived by broker 103 (block 1701). The broker responds by solicitingthe necessary data to verify the identity of the client, and using thedata provided to verify the client's identity (block 1702). Identity ofthe client is preferably verified at login time using techniques similarto those used to verify the identity of the surveillance domainexplained above with respect to FIG. 15, although other or additionalmeasures could be used.

Once the identity of the client is verified, the client is logged on tothe broker and the broker waits for an incoming service request from theclient, represented as the tight loop at block 1703. The delay loop atblock 1703 does not necessarily imply that the client service function434 is continually executing, but it may be placed in an idle state byOS Kernel 401 while awaiting an incoming request. Upon receipt of arequest, the request is processed, represented as the ‘Y’ path fromblock 1703.

If the incoming request from the client is a request to authorize accessto surveillance video produced by a specifically identified surveillancedomain, the ‘Y’ branch is taken from block 1704, and the request toaccess video is processed by querying the database to determine whetherthe requesting client is authorized to access the requested surveillancevideo under current conditions, and generating and transmitting theappropriate authorization or denial in response (represented in FIG. 17as block 1705, and shown in greater detail in FIGS. 18A and 18B). Afterservicing the request, control returns to block 1703 to await the nextrequest.

FIGS. 18A and 18B (herein collectively referred to as FIG. 18) are aflow diagram illustrating in greater detail a process for handling arequest by a client to access surveillance video, according to thepreferred embodiment. In the representation of FIG. 18, actionsperformed by a client 104 requesting access to surveillance video areshown in the leftmost column, actions performed by surveillance videobroker 103 are shown in the second column from left, actions performedby the surveillance domain 102 which produces the requested video areshown in the third column from left, and actions performed by anoptional video distribution service 105 (if present) are shown in therightmost column.

Referring to FIG. 18, a client (previously registered and logged in tobroker 103) transmits a request to the broker to access a specificsurveillance video stream (block 1801). The video access requestincludes any necessary parameters, specifically an identification of thesurveillance domain and, optionally, a specific camera within thatsurveillance domain. The client may have knowledge of specificsurveillance domains and cameras independently of the broker, or it mayfirst request the broker to search its database 403 for surveillancedomains meeting some set of criteria, as explained in greater detailherein with respect to FIG. 17, block 1707, selecting a domain and/orcamera from the set of search results for inclusion in the video accessrequest. In the case of a mobile client, the access request may alsoinclude a client device location, since access may be conditioned on thelocational proximity of the client device. The video access request isreceived by the broker (block 1802).

Upon receipt of the video access request, video access function 435executing in broker 103 queries database 403 to determine whether therequesting client is authorized to access the requested video undercurrent conditions (block 1803). (Alternative embodiments in which thesurveillance domain itself maintains access authorization records and/ormakes the access determination are illustrated as optional blocks1828-1830, and are discussed below.) A client authorized to accesssurveillance video is one who is either (a) identified as a recipient,or (b) a member of a client group identified as a recipient, in therecipient ID field 1204 of an entry 1201 in authorizations table 406which matches the parameters of the received video access request undercurrent condition. This can be determined using standard database querytechniques. For example, a query of the form:

FIND authorizationrecord in AuthorizationsTable WHERE   [clientID ofrequesting client] matches AuthorizationsTable.recptID   AND   [sourcedomain requested] = AuthorizationsTable.domainID AND   [source camerarequested] matches AuthorizationsTable.cameraID   AND   [currentdate/time] > AuthorizationsTable.strtdate/time AND   [current date/time]< AuthorizationsTable.enddate/time AND   [current time] >AuthorizationsTable.dailystrt AND   [current time] <AuthorizationsTable.dailyend AND   [current day] matchesAuthorizationsTable.dayofweekflags AND     [AuthorizationsTable.uf =true OR         [[currentstatus] matches AuthorizationsTable.c1 AND        [currentstatus] matches AuthorizationsTable.c2 AND . . .        [currentstatus] matches AuthorizationsTable.cn AND        [currentstatus] matches AuthorizationsTable.proximity]]would return a record if the client is directly identified as recipient,and a query of the form:

FIND authorizationrecord in AuthorizationsTable WHERE   [clientID ofrequestor] = GrpMembTable.clientID AND   GrpMembTable.groupID =AuthorizationsTable.recptID AND   [source domain requested] =AuthorizationsTable.domainID AND   [source camera requested] matchesAuthorizationsTable.cameraID   AND   [current date/time] >AuthorizationsTable.strtdate/time AND   [current date/time] <AuthorizationsTable.enddate/time AND   [current time] >AuthorizationsTable.dailystrt AND   [current time] <AuthorizationsTable.dailyend AND   [current day] matchesAuthorizationsTable.dayofweekflags AND     [AuthorizationsTable.uf =true OR         [[currentstatus] matches AuthorizationsTable.c1 AND        [currentstatus] matches AuthorizationsTable.c2 AND . . .        [currentstatus] matches AuthorizationsTable.cn AND        [currentstatus] matches AuthorizationsTable.proximity]]would return a record if the client is a member of a group identified asa recipient.

As explained previously, multiple authorization records may exist forthe same surveillance domain to permit different authorizationconditions for different requesting clients, and to allow logical ORs ofmultiple conditions. For example, if it is desired to allow access ifeither condition ci or cj is satisfied, there will be two authorizationrecords in authorization table 406, one corresponding to condition ci(in which the other conditions have wildcard values) and onecorresponding to condition cj (in which the other conditions havewildcard values). If access is authorized for a declared state ofemergency, this condition would typically be expressed in a separateauthorization record in which all other conditions have wildcard values.

If no authorization record is found in the database which matches theparameters of the video access request and the current conditions, the‘N’ branch is taken from block 1804, and the broker transmits a denialof access message back to the requesting client (block 1805). Therequesting client device receives the message and notifies the user,e.g., by display on a display screen (block 1806).

If an authorization record is found which matches the parameters of thevideo access request and current conditions, the ‘Y’ branch is takenfrom block 1804. In this case, video access function 435 generates anaccess token and records the access authorization in accesses table 407of database 403 (block 1807). The access token is a generic designationof any data which the client will use to obtain access to the requestedvideo from a provider of the video, which could be the surveillancedomain itself or some other provider, such as video distribution service105. For example, a token may be simply a password, but it mayalternatively be a more complex data structure some or all of which maybe encrypted, and which may include access information such as an IPaddress of the provider, port, protocol, information. Preferably, thetoken is a one-time authorization, at least a part of which is unique tothat particular access authorization, and can not be used by otherclients or re-used by the same client. The token may include a timeduration of the authorization (derived either from end date/time field1206, daily end field 1209, duration field 1213, or other means, such asa default maximum value), although time duration information may betransmitted separately from the broker to the provider without inclusionin the token.

After generating the token, the broker transmits an access authorizationnotice to the applicable surveillance domain to notify it that accesshas been authorized for the requesting client (block 1808). This noticepreferably includes the token, allowing the surveillance domain torecognize the token when the client makes its request. The surveillancedomain receives and stores the access authorization notice (block 1809).

At this point, two alternative paths of control are represented in FIG.18. In a first alternative embodiment, represented by block 1810-1818,the surveillance video is provided directly by surveillance domain 102.In this case, blocks shown in dashed lines are not executed. In thesecond alternative, the video is provided by video distribution service105.

Specifically, in the first alternative embodiment, the broker transmitsthe authorization, including the token, to the client (block 1810).Since the client does not necessarily know the address of the applicablesurveillance domain, the authorization notice includes this information(which may be contained in the token). The broker has no further roleafter transmitting the various authorizations, and does not serve as aconduit for the video stream itself. The client receives theauthorization and token (block 1811), and transmits an access requestcontaining the token to the surveillance domain at the address provided(block 1812). The address used to access the surveillance domain is notnecessarily the same as that used by broker to communicate with localcontrol system 202. Local control system 202 may in fact be multiplesystems having multiple addresses, and a separate address may be usedfor accessing the video stream.

The surveillance domain receives the access request and token (block1813). It verifies the token based on information earlier receiveddirectly from the broker at block 1809 (block 1814). Assuming the tokenis verified, the surveillance domain then begins transmission of videoto the client (block 1815). The client receives the transmitted videoand displays it to a user on a display screen at the client (block1816).

Typically, at some point thereafter the client will decide to stopwatching the video stream, and disconnect or otherwise indicate to thesurveillance domain that video transmission may be stopped (block 1817).The surveillance domain receives this indication, and halts transmissionof the video stream (block 1818). Alternatively, transmission of a videostream may be halted by action of the surveillance domain, e.g., uponexpiration of the time duration of any authorization (not shown in FIG.18).

In the second alternative embodiment employing video distributionservice (VDS) 105, the broker not only transmits the authorization tothe surveillance domain (block 1808) and the client (block 1810), butalso transmits the authorization, including the token, to the videodistribution service (block 1819). The video distribution servicereceives and stores the authorization (block 1820.

Depending on the implementation, a surveillance domain employing a videodistribution service may be continually providing video to the VDS atall times, or may do so only on demand. If the surveillance domain isnot already providing the video to the VDS, the surveillance domaincommences transmission of the video to the VDS responsive to receivingthe notification of authorization to access video (block 1821). The VDSreceives the video stream and buffers it (block 1822). The VDS may alsoarchive the video for more permanent storage.

Blocks 1810, 1811 and 1812 are performed in the second alternativeembodiment as in the first, except that the address for accessing videoprovided by the broker is the address of the VDS, and the requesttransmitted by the client at block 1812 is to the VDS. Blocks 1813-1815are therefore not performed in the second alternative embodiment. TheVDS receives the request from the client, including token (block 1823),verifies the token (block 1824), and transmits the requested videostream to the client (block 1825).

The client receives the video transmitted from the VDS and displays itto a user on a display screen at the client (block 1816). When theclient thereafter disconnects or otherwise indicates that videotransmission may be stopped (block 1817), this is communicated to theVDS, which then halts transmission (block 1826). As in the case of videosupplied directly by the surveillance domain, the VDS may act by itselfto halt transmission upon expiration of the time duration of anyauthorization (not shown in FIG. 18). If the video stream is beingprovided by the surveillance domain to the VDS only on demand, ratherthan continuously, the VDS will also notify the surveillance domain thatit no longer needs the video (block 1827), causing the surveillancedomain to halt transmission of video to the VDS.

The video access authorization procedure described above with respect toFIG. 18 is specifically intended for use in authorizing real-time accessto a surveillance video stream. However, essentially the same proceduremay be used for authorizing access to a previously recorded and archivedvideo stream, although if an optional VDS is used, it would be likelythat the archived video is already stored in the VDS, and hence blocks1821, 1822 and 1827 would not be necessary. Furthermore, the criteriaused to authorize access to archived video may be different. Forexample, the use of an emergency state condition to authorize access tosurveillance video in the event of a declared emergency is useful forreal-time video, but would have less utility for archived video. Toaccount for these differences, one or more additional flags (not shown)might be included in each video authorization record in authorizationstable 406 to indicate whether the corresponding authorization applies toreal time or archived video or both. In one variation of the preferredembodiment, conditional access applies only to real-time video, and onlya client authorized to unconditional access, or manually given specialpermission, may access the archived video.

Referring again to FIG. 17, if the incoming request from the client issomething other than a request to access surveillance video, the ‘N’branch from block 1704 is taken. A client may request the broker tosearch its database for surveillance domains (or specific cameras)represented as the ‘Y’ branch from block 1706. A client may request sucha search preliminary to a request to access video from one of thedomains as described above with respect to FIG. 18, or to find potentialunauthorized domains which the client may wish to contact directly andrequest that the domain include the client in the authorizationsrecorded in the broker, or for some other reason. In this case, clientservice function 434 queries database 403 to find applicablesurveillance domains or cameras (block 1707). I.e., the client ispresented with one or more interactive screens (e.g. interactive webpages) in which to enter the parameters of a search. One or more commonsearches could be presented to the client in a prepared format, with anoption to perform a custom search should the client desire it, using anyconventional query technique to specify search terms. The client servicefunction then invokes query engine 422 to perform the search inaccordance with the specified parameters, returns search results to theclient, and returns to the idle loop at block 1703. For example, aclient may wish to search for all surveillance domains to which it isauthorized. Alternatively, a client may wish to search for allsurveillance domains located within some specified distance from theclient. Alternatively, a client may wish to search for a camera at aspecific location. Other searches are possible. In returning searchresults, client service function 434 will filter (i.e., remove from theresults) any surveillance domain which has its privacy flag 805 set ifthe requesting client is not actually authorized to access videoproduced by the surveillance domain.

If the incoming request is neither a request to access surveillancevideo nor a request to search domains or cameras (the ‘N’ branch fromblock 1706), a client may further request to declare a state ofemergency in order to enable emergency services personnel to accesscertain surveillance video, represented as the ‘Y’ branch from block1708. In this case, client service function 434 verifies the authorityof the requesting client and handles the declaration of emergency state(represented in FIG. 17 as block 1709, and shown in greater detail inFIG. 19). After servicing the request, control returns to block 1703 toawait the next request.

FIG. 19 is a flow diagram illustrating in greater detail a process forhandling a request by a client to declare a state of emergency,according to the preferred embodiment. In the representation of FIG. 19,actions performed by a client 104 requesting the declaration ofemergency state are shown in the leftmost column, actions performed bysurveillance video broker 103 are shown in the middle column, andactions performed by certain surveillance domains 102 which producevideo are shown in the right column.

Referring to FIG. 19, a client (previously registered and logged in tobroker 103) transmits a request to the broker to declare a state ofemergency (block 1901). The request may include certain parameters, suchas a reason for the emergency, a duration of the emergency, and soforth. If no duration is specified in the request, a default durationmay be assigned. The decision to declare the emergency is made by ahuman user, which would typically be an authorized public official.Examples of causes for declaring an emergency could include a fire, anatural disaster such as a tornado, earthquake flood, etc., suspectedcriminal activity, chemical spill or radiation leak, gas leak, and soforth. An emergency could be restricted to a specific geographical area(e.g. a city block), in which case the parameters of the emergency wouldinclude the geographical limitation. In one aspect, an emergency isbased on an alert condition detected by a nearby surveillance domain;for example, a fire alarm detected at a first surveillance domain couldcause an emergency to be declared for neighboring surveillance domains,enabling access to their surveillance video by firefighters or otheremergency responders. The emergency request is received by the brokerand stored (block 1902).

Upon receipt of the request to declare an emergency, client servicefunction 434 verifies the authority of the requesting client to declarethe emergency state (block 1803). This can be performed by accessing thecorresponding entry 501 in clients table 410 and verifying the state ofemergency authorization flag 515, or verifying the state of anyequivalent or substitute data structure. It will be appreciated thatmore elaborate mechanisms might be used for verifying authority todeclare an emergency, but preferably, in order to respond in theshortest possible time, any such mechanism is pre-authorized in datamaintained in the broker, and does not require manual confirmation atthe time of the emergency.

In the client's authority to declare a state of emergency can not beverified, the ‘N’ branch is taken from block 1904, and the brokertransmits a denial of emergency state request message back to therequesting client (block 1905). The requesting client device receivesthe message and notifies the user, e.g., by display on a display screen(block 1906).

If client authority to declare an emergency is verified, the ‘Y’ branchis taken from block 1904. In this case, client service function 434records emergency state in a suitable temporary data structure, theemergency state including any necessary parameters such as duration andgeographical limitations (block 1907). Since the emergency state ispreferably of limited duration, a timer is set to expire upon theexpiration of the emergency state. The client service function furtherrecords the emergency state grant as an event in events table 405 ofdatabase 403, from which it may be subsequently queried and examined foraudit or other purposes.

An emergency state having been declared responsive to the client'srequest, the client service function transmits an acknowledgment of theemergency state back to the requesting client (block 1908). Theacknowledgment is received by the requesting client and the user isnotified (block 1909).

The client service function further queries the database to determinewhich surveillance domains might be affected by the declaration ofemergency state (block 1910). An affected surveillance domain is onewhich has one or more video access authorization records inauthorizations table 406 which authorize access conditional upon theexistence of an emergency state, and which falls within the geographicallimitations, if any, of the emergency state. A notification of thedeclaration of the emergency state is transmitted to each such affectedsurveillance domain (block 1911), and the affected surveillance domainstores and/or displays the notification to a user (block 1912). Theclient service function returns to the wait loop at block 1703 of FIG.17 after transmitting notifications in block 1911.

After some time has elapsed, the emergency expires, which is detected byexpiration of the timer set at the time the emergency was declared(block 1913). The client service function reset its state to remove theemergency, preventing any further grant of access based upon theemergency. The record of the emergency is not deleted from events table405, but is kept there as a permanent record of the emergency. Theclient service function transmits a notice that the emergency hasexpired to the requesting client and to each affected surveillancedomain (block 1914). The notice is received by the client and displayedto the user (block 1915), and received by the surveillance domains andstored or displayed as appropriate (block 1916).

Referring again to FIG. 17, if the incoming request from the client isneither a request to access surveillance video, nor a request to searchsurveillance domains, nor a request to declare a state of emergency, the‘N’ branch from block 1708 is taken. A client may further request thebroker to examine logs of previous events or previous grants of access,represented as the ‘Y’ branch from block 1710. In this case, clientservice function 434 queries database 403 to find applicable events orgrants of access (block 1711). Events are stored in events table 405,and include any non-scheduled event which might affect authority toaccess video, such as a declaration of emergency condition or an alarmevent issued by a surveillance domain. Grants of video access authorityare recorded in accesses table 407. Preferably, the client servicefunction presents the client with one or more interactive screens forinput, and standard database search techniques may be used to define theparameters of the search of events table 405 or accesses table 407, asthe case may be. The database is searched accordingly, search resultsare returned to the client, and the client service function returns tothe wait loop at block 1703. For reasons of privacy, there may bedefault rules in place preventing a client from seeing certain events oraccesses in which it has no apparent legitimate interest. E.g., a clientmay view its own accesses or accesses to a surveillance domain to whichit has access, but not accesses by others to a surveillance domain towhich the client has no access.

If the request from the client is neither a request to accesssurveillance video, nor a request to search surveillance domains, nor arequest to declare a state of emergency, nor a request to examinepreviously logged events or accesses (the ‘N’ branch from block 1710), aclient may further request to add, delete or otherwise edit alertnotification filter records for alerts notifications it wishes toreceive, represented as the ‘Y’ branch from block 1712. In this case,the client service function presents one or more interactive selectionand editing screens to a client, allowing the client to search for aprevious notification filter record (stored in filters table 409), editor delete a previous notification filter record when found, and/orgenerate a new notification filter record (block 1713). This ispreferably accomplished by presenting one or more interactive web pagesto a user executing a browser application within the client device. Anychanges are stored in one or more appropriate entries in filters table409, and the client service function returns to the idle loop at block1703.

If the request from the client is neither a request to accesssurveillance video, nor a request to search surveillance domains, nor arequest to declare a state of emergency, nor a request to examinepreviously logged events or accesses, nor a request to edit alertnotification filters (the ‘N’ branch from block 1712), a client mayfurther request to edit client group data by adding or deleting a group,adding or deleting members from a group, or otherwise editing thegroup's profile data, represented as the ‘Y’ branch from block 1714. Inthis case, if the group is an existing group, client service functionverifies authority of the requesting client to alter group data (block1715). The client has authority if the client is identified as thegroup's administrator in the admin ID field 603 of the correspondinggroup record 601 in groups table 414 (or if the client is a member of agroup identified as the group's administrator in the admin ID field). Ifauthority is verified (or if the group is a new group, requiring noverification of authority), the client is presented with one or moreinteractive editing screens allowing it to edit group information, inparticular to add or delete members from the group (block 1716). Theclient service function saves any changes in the database and returns tothe idle loop at block 1703.

If the request from the client is neither a request to accesssurveillance video, nor a request to search surveillance domains, nor arequest to declare a state of emergency, nor a request to examinepreviously logged events or accesses, nor a request to edit alertnotification filters, nor a request to edit client group data (the ‘N’branch from block 1714), a client may further request to edit otherprofile data, such as data defining the parameters of the client,represented as the ‘Y’ branch from block 1717. In this case, the clientservice function may present one or more interactive selection andediting screens to a client allowing the client to edit the desiredinformation (block 1718). Alternatively, block 1718 may representcertain information provided or edited by an automated program executingin a client device. In particular, in the case of a mobile clientdevice, it may be desirable to periodically update the location of theclient device by automatically providing such information to the brokervia a series of requests, either at pre-determined intervals or when acurrent location becomes sufficiently distant from a previously suppliedlocation. Any changes are stored in appropriate location(s) in database403, and the client service function returns to the idle loop at block1703.

If the incoming request is none of the above described requests (the ‘N’branch from block 1717), a client may further request other services(not including logout), represented as the ‘N’ branch from block 1719.In this case, the client service function handles the request asappropriate (block 1720). Other request may include, e.g., requests formaintenance and diagnostic services, accounting and billing matters, orany other service which might be useful. After servicing the request,the client service function returns to the idle loop at block 1703. Ifthe request is a logout request (the ‘Y’ branch from block 1719), theclient service function terminates and the client is logged out.

Although a series of steps has been described above as a preferredembodiment, it will be appreciated that many variations of a process foraccessing surveillance video and associated tasks are possible, and inparticular that some steps may be performed in a different order, thatdifferent data structures may be used, and/or that different hardware orsoftware resources may be employed to perform functions describedherein.

In the preferred embodiment described above, a surveillance video brokermaintains video authorization records on behalf of each of multiplesurveillance domains and automatically grants video access to requestingclients based on the pre-existing authorizations in the videoauthorization records. However, it would alternatively be possible forthe broker to forward each access request from a client to thesurveillance domain, allowing the surveillance domain to make thedetermination whether to grant access to its video. In this alternativeembodiment, the broker registers and authenticates the clients asdescribed above. A client desiring access to video logs in to the brokerand submits its request to the broker as described above, but thebroker, rather than determine whether to authorize access, forwards therequest to the surveillance domain for action (as shown in optionalblock 1828 in FIG. 18), which receives the request (optional block1829). The surveillance domain maintains such records as are necessaryto make a determination, with respect to each requesting client andcurrent conditions, whether to authorize access. These records may besimilar to those described above and maintained by the broker, except ofcourse that a surveillance domain will only maintain authorizationrecords for itself and not for other surveillance domains. Thesurveillance domain could then make a determination whether to grantaccess to the requested video based upon its authorization records andthe current state of any conditions which might affect conditionalauthorization (optional block 1830). If the surveillance domaindetermines to grant the request, it preferably returns an appropriatemessage back to the broker (the return path to block 1804), whichcommunicates with the client. The surveillance domain couldalternatively communicate directly with the client at this point. As afurther alternative, the surveillance domain could maintain one or moreauthorization records, and transmit the applicable record(s) or datatherefrom to the broker upon request responsive to a client request foraccess (the return path from block 1830 to block 1803), allowing thebroker to them make the determination whether to grant access based uponthe transmitted authorization record data and the current state of anyapplicable conditions. These alternative embodiments may have someadvantages in terms of providing maximum flexibility for thesurveillance domains to make access determinations and reducing thecomplexity of the broker, but it is believed there are countervailingadvantages to the technique described as a preferred embodiment, e.g.,avoiding duplication of decision-making software, uniformity of accessdeterminations, ability of the broker to hide unauthorized domains fromclients, etc.

In a similar vein, although in the preferred embodiment the brokermaintains notification records for determining whom to notify in theevent of an alert, in an alternative embodiment the alert message itselfcould carry notification parameters defining whom to notify. If anyfiltering of the alert message is to be performed, this could be done inthe client. I.e., the broker could transmit alert messages to allclients authorized by the issuing surveillance domain to receive them,and the client could determine whether to display the alert to a user.Furthermore, while in the preferred embodiment the broker handles bothaccess to surveillance video and transmission of alert notificationsgenerated by a surveillance domain, the broker might alternativelyhandle only the access to surveillance video, alert notifications beinghandled by a separate mechanism.

In the preferred embodiment described above, the authorization recordstake the form of records in a relational database which are accessedusing general-purpose database search tools. However, in general anauthorization record might be anything which records the informationnecessary to make an access authorization determination, and need not bea record in a relational database. For example, an authorization recordmight be nothing more than one or more lists of authorized clients, oneof which might indicate unconditional access, while one or more othersindicate access under respective conditions. Other data structures wouldalternatively be possible. Notification records or notification filtersmight similarly take other forms.

In one or more variations of a preferred embodiment, one or more of theclients are employees or other agents of public authorities acting as oron behalf of emergency responders (herein designated a “public authorityemergency responder”), such as firefighters, police, emergency medicalpersonnel, or dispatchers or other support personnel. Access tosurveillance video generated by a privately-controlled surveillancedomain is automatically authorized as described herein upon theoccurrence of any of various pre-specified events, such as a specifiedalarm condition. For example, firefighters may be automaticallyauthorized upon the occurrence of a fire alarm; police may beautomatically authorized upon the occurrence of an intruder alarm, storeclerk alarm, or similar indication of criminal activity, or any ofvarious events in which police may take an interest, for example a fire.Additionally, such public authority emergency responders may beauthorized to access video from a privately-controlled surveillancedomain upon declaration of an emergency state, even in the absence of aspecific alarm. For example, criminal activity often occurs without analarm sensor being tripped, and may be detected by visual observation orother means, causing an appropriate person, such as an authorized policeofficial, to declare an emergency. Additionally, such authorization maybe conditioned in whole or in part on the location of the publicauthority emergency responder. This capability substantially increasesthe ability of emergency personnel to access surveillance video in realtime, since it does not first require manual authorization by a personhaving access control of a privately-controlled surveillance domain, whomay be unavailable.

As a further alternative embodiment, the concept of “emergency access”can be broadened to include any arbitrary client or client group, oreven all clients, each of whom is allowed to assert a limited“emergency” condition on its own behalf in order to obtain access tosurveillance video. Such limited emergency access is still governed bythe surveillance domain's access grants, i.e., the surveillance domainwould specify such access rights in one or more access authorizationrecords, as described herein. Such a limited emergency is an “event”which is logged for audit as described herein.

As a further alternative, the concept of a “group” could be applied tosurveillance domains as well as clients. I.e., groups of surveillancedomains could be defined in the same way that groups of clients aredefined, and access rights could be granted by a defined group ofsurveillance domains to any surveillance video produced by any member ofthe group. Such a group definition might make management of multiplesurveillance domains easier where the multiple domains have commonownership or other characteristics.

In general, the routines executed to implement the illustratedembodiments of the invention, whether implemented as part of anoperating system or a specific application, program, object, module orsequence of instructions, including a module within a special devicesuch as a service processor, are referred to herein as “programs” or“control programs”. The programs typically comprise instructions which,when read and executed by one or more processors in the devices orsystems in a computer system consistent with the invention, cause thosedevices or systems to perform the steps necessary to execute steps orgenerate elements embodying the various aspects of the presentinvention. Moreover, while the invention has and hereinafter will bedescribed in the context of fully functioning computer systems, thevarious embodiments of the invention are capable of being distributed asa program product embodied in non-transitory computer-readable media,and the invention applies equally regardless of the form ofdistribution. Examples of non-transitory computer-readable mediainclude, but are not limited to, volatile and non-volatile memorydevices, floppy disks, hard-disk drives, CD-ROM's, DVD's, and magnetictape, it being understood that these examples are not exhaustive.Examples of non-transitory computer-readable media are illustrated inFIG. 3 as system memory 302 and data storage devices 325-327.

Unless inconsistent with the invention or otherwise qualified herein,computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in anon-transitory computer-readable medium that can direct a computer orother programmable data processing apparatus to function in a particularmanner, such that the instructions stored in the non-transitorycomputer-readable medium produce an article of manufacture includinginstruction means which implement the function/act specified in theflowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide processes for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Although a specific embodiment of the invention has been disclosed alongwith certain alternatives, it will be recognized by those skilled in theart that additional variations in form and detail may be made within thescope of the following claims:

What is claimed is:
 1. A surveillance video broker for managing accessto surveillance video by multiple clients, comprising: at least oneprocessor; a memory; at least one network adapter for communicating witha plurality of surveillance domains and a plurality of clients over atleast one network; a video access controller embodied as a plurality ofinstructions recordable in said memory and executable on said at leastone processor, said video access controller including: a registrationfunction for independently registering a plurality of surveillancedomains and a plurality of clients, each surveillance domain producingat least one respective surveillance video stream, each surveillancedomain comprising a respective set of one or more video cameras operatedby a respective operating entity common to all video cameras of therespective set, wherein at least some of the video surveillance domainsare operated by different respective operating entities; an accessauthorization function which, responsive to a request from a clientregistered by said registration function to access in real time videoproduced by a target surveillance domain of said plurality ofsurveillance domains, causes at least one access authorization recordcorresponding to the target surveillance domain to be accessed todetermine whether surveillance video access is authorized, and ifsurveillance video access is authorized, automatically grants real-timeaccess to the requested video, wherein each surveillance domainregistered by said registration function independently defines arespective set of clients authorized to access surveillance videoproduced by the respective surveillance domain in at least onecorresponding access authorization record, each authorization recordcorresponding to a single respective surveillance domain of saidplurality of surveillance domains, each authorization record grantingaccess to one or more respective clients independently of association ofthe one or more respective clients with the respective operating entityof the corresponding surveillance domain and independently of whetherthe one or more clients have been granted access to one or more othersurveillance domains, wherein a first client is granted access to afirst set of video surveillance domains containing multiple videosurveillance domains but fewer than all of said plurality of videosurveillance domains by respective authorization records correspondingto each surveillance domain of the first set, and wherein a secondclient is granted access to a second set of surveillance domainscontaining some but not all fo the sources to which the first client hasaccess by respective authorization records corresponding to eachsurveillance domain of the second set, wherein at least oneauthorization record specifies conditional authorization based on arespective at least one specified condition.
 2. The surveillance videobroker claim 1, wherein said authorization records are stored in saidsurveillance video broker.
 3. The surveillance video broker of claim 1,wherein multiple respective authorization records are defined for eachsurveillance domain, each authorization record of the multiplerespective authorization record for a first surveillance domain grantingaccess to a respective set of at least one client, at least some of therespective sets of at least one client and at least one condition ofaccess being different.
 4. The surveillance video broker claim 1,wherein at least one authorization record specifies unconditional accessauthorization for one or more respective clients.
 5. The surveillancevideo broker claim 1, wherein at least one authorization recordspecifies conditional authorization based on an alarm condition detectedby an automatic sensor in the corresponding video surveillance domain.6. The surveillance video broker method of claim 1, wherein at least oneauthorization record specifies conditional authorization based on ascheduled event.
 7. The surveillance video broker of claim 1, wherein atleast one authorization record specifies conditional authorization basedon physical proximity of the requesting client to the correspondingsurveillance video domain.
 8. The surveillance video broker of claim 1,wherein at least one authorization.
 9. The surveillance video broker ofclaim 1, further comprising: an event notification function which,responsive to an event alert notification request from a surveillancedomain identifying an alert event detected at the surveillance domainfor which notification to clients is requested, causes at least onenotification record corresponding to the surveillance domain for whichnotification to clients is requested to be accessed to determine a setof clients to receive notification of the alert event, and transmits analert notification to each client of the set, wherein each notificationrecord corresponds to a single respective surveillance domain of saidplurality of surveillance domains and independently specifies on behalfof he corresponding surveillance domain at least one respective clientauthorized to receive notification of one or more respective alertevents generated by the corresponding surveillance domain, each saidnotification record specifying at least one respective client authorizedto receive notification independently of whether the at least onerespective client is authorized by said authorization records to accesssurveillance video generated by the corresponding surveillance domain.10. A method for authorizing access to video surveillance data,comprising: registering a plurality of clients in a video broker dataprocessing system; registering a plurality of video surveillance domainsin said video broker data processing system, each surveillance domaincomprising a respective set of one or more video cameras operated by arespective operating entity common to all video cameras of therespective set, wherein at least some of the video surveillance domainsare operated by different respective operating entities; recording aplurality of authorization records, each authorization record associatedwith a single respective video surveillance domain of said plurality ofvideo surveillance domains and independently specifying authorization onbehalf of the respective video surveillance domain with which it isassociated to allow at least one respective client real-time access toat least some surveillance video generated by the respective videosurveillance domain with which it is associated, each authorizationrecord granting access to one or more respective clients independentlyof association of the one or more respective clients with the respectiveoperating entity and independently of whether the one or more respectiveclients have been granted access to one or more other video surveillancedomains, wherein a first client is granted access to a first set ofvideo surveillance domains containing multiple video surveillancedomains but fewer than all of said plurality of video surveillancedomains by respective authorization records associated with each videosurveillance domain of the first set, and wherein a second client isgranted access to a second set of video surveillance domains containingsome but not all of the sources to which the first client has access byrespective authorization records associated with each video surveillancedomain of the second set; receiving in said video broker a plurality ofvideo access requests, each request requesting real-time access by arespective client to at least some video produced by a respectivesurveillance domain; responsive to each said video access request,automatically using the one or more authorization records associatedwith the respective surveillance domain to which the respective clientrequested access to determine whether the respective client isauthorized to access the requested video; responsive to determining thatthe respective client is authorized to access the requested video of avideo access request, automatically granting real-time access by therespective client to the requested video.
 11. The method of claim 10,wherein said recording in said video broker data processing system aplurality of authorization records comprises recording a first and asecond authorization record specifying authorization on behalf of afirst video surveillance domain, the first authorization record grantingaccess to a first set of at least one client and the secondauthorization record granting access to a second set of at least oneclient, said first and second sets being different, said firstauthorization record specifying conditional access based on a first atleast one specified condition, said second authorization record notspecifying conditional access based on said first at least one specifiedcondition.
 12. The method of claim 11, wherein said second authorizationrecord specifies unconditional access by said second set of at least oneclient.
 13. The method of claim 10, wherein at least one authorizationrecord specifies conditional authorization based on an alarm conditiondetected by an automatic sensor in the respective video surveillancedomain.
 14. The method of claim 10, wherein at least one authorizationrecord specifies conditional authorization based on a scheduled event.15. The method of claim 10, wherein at least one authorization recordspecifies conditional authorization based on physical proximity of therequesting client to the surveillance video domain.
 16. The method ofclaim 10, wherein at least one authorization record specifiesconditional authorization based on a declared state of emergency. 17.The method of claim 10, wherein automatically granting real-time accessby the respective client to the requested video comprises generating andtransmitting a respective access token to the respective client, theaccess token being used by the client to obtain real-time access to therequested video directly from a provider of the requested video, withouttransmitting the requested video through said video broker dataprocessing system.
 18. The method of claim 17, wherein said provider ofthe requested video is a video distribution service which distributesvideo received from a plurality of surveillance domains.
 19. A methodfor operating a data processing system of a surveillance video broker,comprising: registering a plurality of clients in a data processingsystem of a surveillance video broker; registering a plurality of videosurveillance domains in said data processing system of said surveillancevideo broker, each video surveillance domain comprising a respective setof one or more video cameras operated by a respective operating entitycommon to all video cameras of the respective set, wherein at least somevideo surveillance domains are operated by different respectiveoperating entities; recording in said data processing system a pluralityof authorization records, each authorization record associated with asingle respective video surveillance domain of said plurality of videosurveillance domains and independently specifying authorization onbehalf of the respective video surveillance domain with which it isassociated to allow at least one respective client real-time access toat least some surveillance video generated by the respective videosurveillance domain with which it is associated, each authorizationrecord granting access to one or more respective clients independentlyof association of the one or more respective clients with the respectiveoperating entity and independently of whether the one or more respectiveclients have been granted access to one or more other video surveillancedomains; recording in said data processing system a plurality ofnotification records, each notification record associated with a singlerespective video surveillance domain of said plurality of videosurveillance domains and independently specifying on behalf of therespective video surveillance domain with which it is associated atleast one respective client authorized to receive notification of one ormore respective alert events generated by the respective videosurveillance domain, each said notification record specifying at leastone respective client authorized to receive notification independentlyof whether the at least one respective client is authorized by saidauthorization records to access surveillance video generated by therespective video surveillance domain with which the respectivenotification record is associated; receiving in said data processingsystem of said surveillance video broker a plurality of alert eventnotification requests, each alert event notification request beingreceived from a respective video surveillance domain and identifying analert event detected at the respective surveillance domain for whichnotification to clients is requested; responsive to receiving each saidalert event notification request, accessing each notification recordassociated with the respective surveillance domain from which therespective alert event notification request was received s to determinein said data processing system a respective set of clients to receivenotification of the respective alert event, and transmitting an alertnotification to each client of the respective set; receiving in saiddata processing system of said surveillance video broker a plurality ofvideo access requests, each request requesting real-time access by arespective client to at least some video produced by a respectivesurveillance domain; responsive to each said video access request,automatically using the one or more authorization records associatedwith the respective surveillance domain to which the respective clientrequested access to determine whether the respective client isauthorized to access the requested video; responsive to determining thatthe respective client is authorized to access the requested video of avideo access request, automatically granting real-time access by therespective client to the requested video.
 20. The method of claim 19,wherein at least one authorization record specifies conditionalauthorization based on at least one respective specified condition.